https://pandorawiki.org/api.php?action=feedcontributions&user=Vinipsmaker&feedformat=atomPandora Wiki - User contributions [en]2024-03-28T16:54:10ZUser contributionsMediaWiki 1.32.0-alphahttps://pandorawiki.org/index.php?title=New_PND_format&diff=29541New PND format2014-02-20T22:09:41Z<p>Vinipsmaker: multiple binaries apps decided</p>
<hr />
<div>The current PND format has some shortcomings as listed below. This page should serve as a discussion page/white board for how the format could be improved.<br />
<br />
''' Proposal for discussion, this is just some opening shots '''<br />
<br />
== Current situation ==<br />
The current (ISO-based) PND format has the following shortcomings:<br />
<br />
* It currently uses ISO, SquashFS, and other filesystems... This means that there's no standard to follow, and libpnd needs to carry around support for a multitude of file systems.<br />
* It (often, but not always) uses the [http://en.wikipedia.org/wiki/ISO_9660 ISO file system] which is inefficient at storing data, because:<br />
** It contains duplicate headers for each file entry[http://users.telenet.be/it3.consultants.bvba/handouts/ISO9960.html] (one version with big-endian integers, one version with little-endian integers[http://en.wikipedia.org/wiki/Endianness]) which leads to many kilobytes of wasted space.<br />
** Its file tables are fixed-size, and there are therefore limitations on how many folders you can have, what names you can give your files, how big your files can be etc. For instance, files can only have names that are max 31 characters long, all upper-case, and limited to the ASCII character encoding, and an ISO does only support a folder depth of 8[http://www.on-time.com/rtos-32-docs/rtfiles-32/programming-manual/iso-9660/]. To support more, the Joliet file system extension is needed (or isofs won't recognize all file paths), and the Joliet header can be placed practically anywhere in the file, which means additional seek times.<br />
** It needs various file system extensions to behave correctly, like for example [http://en.wikipedia.org/wiki/Rock_Ridge Rock Ridge Interchange Protocol] and [http://en.wikipedia.org/wiki/Joliet_(file_system) Joliet]. Without them, it becomes pretty useless as demonstrated above, and with them, it becomes difficult to read by a tool (You can only use isofs! There are no other tools out there for programmers to use!).<br />
** It's very difficult to use since you need special ISO making tools to create the image. ISO is a relatively rare format if you're not technically inclined.<br />
** If you want to add or remove files from it, it's near impossible if you use a compact ISO file, since there's no way that you can "expand" an existing ISO file.<br />
* The PND header data is at the end (or in the middle of the file if a screenshot is included) which makes it impossible for e.g. libmagic to recognize the PND file. It will instead recognize it as an ISO file. Having the header data at the end also makes it take a very long time to find the data, making tools and the libpnd library very inefficient.<br />
* The PND file uses a custom XML format for its metadata. There's no reason to do this, especially since the established ".desktop" file format fills exactly the same function.<br />
(note: ".desktop" does not contain all metadata we have wanted over time, but it's of course possible to add extensions à la "X-Pandora-Whatever=xyz")<br />
* There's no "index" for the PND file. The whole file has to be scanned (albeit backwards) to find a PXML file, and there's a big chance for false positives etc.<br />
* Data is just appended linearly to the file so there's no order. If the format is to be extended (to e.g. include an icon file after the screenshot file), should the data just be appended as well?<br />
* UTF-8 is strictly the only encoding that is supported. If you make your PXML on a Windows machine, it won't work.<br />
<br />
== Proposed revisions ==<br />
=== Step 1: File system ===<br />
The file system for PNDs should be replaced by the ''uncompressed'' ZIP archive format. ZIP has the advantage that it's incredibly compact, and uncompressed ZIP makes it possible to read data from the file without having to do any decompression.<br />
<br />
ZIP files are mountable using various implementations of zipfs, and most of these implementations won't store files in memory when they are being read if the ZIP file is uncompressed.<br />
<br />
=== Step 2: Metadata ===<br />
PND files should no longer require special tools for them to be created. Therefore, since ZIP files support random access on files, it shouldn't be necessary to append/prepend metadata to the file. The user simply includes the "PXML.xml" and "preview.png" files inside of the ZIP, and any tools that need information about the PND can simply go to the central directory of the ZIP file (or use a simple ZIP library to do it for them) and get the location of the file inside of the ZIP. This should also dramatically decrease loading times for PNDs.<br />
<br />
Note; performance testing is needed; at the time we started, zip was enormously slower than plain ISO; with driver changes, it may or may not be so, hence our adopted multiple-filesystem-type system. zipfs is one possible option.<br />
<br />
=== Step 3: Metadata format ===<br />
Current PND files include so-called "PXML.xml" files. These files have a custom XML format that has a strange structure.<br />
<br />
These files should be replaced by ".desktop" files. A PND can contain one or more ".desktop" files in its root directory that specify how an application should be launched. PND tools simply use all ".desktop" files they can find in the PND when creating launchers for the contents of the PND.<br />
<br />
== Advantages ==<br />
* There don't have to be ''any'' special tools for reading PND files. The package can be run on any platform using any programming language that can read ZIP files.<br />
* We can use existing facilities to manage the launching of applications. The ".desktop" files can basically be copied without modification into standard locations of the system, and all launchers will become aware of them.<br />
* Reading PND files becomes easier (because of better tool support... sorry, but you can't link to libpnd in all programming languages) and quicker (becuase of the central directory for random file access).<br />
<br />
=== Benchmarks ===<br />
A benchmark was carried through to measure the performance of various package implementations.<br />
The compared systems were:<br />
* ZIP-based uncompressed packages using fuse-zip (a zipfs implementation) to read the files<br />
* ZIP-based compressed packages also using fuse-zip.<br />
* ISO8859-based packages using isofs<br />
* CramFS-based packages.<br />
<br />
A total of 9 files were generated for the test. They have sizes 1, 2, 3, 4, 16, 32, 64, 128 and 256 MB, and were generated via "dd if=/dev/urandom of=$file bs=1M count=$size".<br />
<br />
Because the files contain random data, they did not compress very well. The resulting files were:<br />
* "zipcompressed.zip" 511.1 MB<br />
* "zipuncompressed.zip" 511.0 MB<br />
* "iso.iso" 511.3 MB<br />
* "cramfs.image" 95.4 MB (WOW!)<br />
<br />
Some notes:<br />
* No write tests were done for obvious reasons<br />
* No random access tests were done since the Pandora will use SD cards and thus not get penalized from random access, and that's not what we're interested in anyways.<br />
* Before each test, the commands "sync; echo 3 > /proc/sys/vm/drop_caches" were run.<br />
<br />
==== First test: linear read time ====<br />
Command: "time cat mountpoint/* > /dev/null"<br />
<br />
* ISO: 8.119 seconds<br />
* CramFS: 1.488 seconds (WOW!)<br />
* Zip (compressed): 8.535 seconds<br />
* Zip (uncompressed): 8.290 seconds<br />
<br />
==== Second test: RAM usage ====<br />
-- TODO, have to find a good method of measuring this --<br />
<br />
== Remaining issues ==<br />
* How should libpnd tell the difference between old and new PND files? Should it depend on the "file" tool, should it run its own recognition algorithm, or what? Should the extension be changed to e.g. ".box" (Which was proposed in a thread)?<br />
* How to make sure that the ZIP files are uncompressed? Should we provide a script that "uncompresses" an ordinary ZIP file?<br />
* Is cramfs better than ZIP?<br />
** cramfs cannot be uncompressed like ZIP can<br />
** If you compare compressed ZIP and cramfs, cramfs is more efficient.<br />
** cramfs is harder to use than ZIP (for the developer).<br />
** cramfs ''needs'' to have all of its opened files in memory.<br />
<br />
== Upgrade path from the old PND format ==<br />
A simple script can be written that extracts the old PND file, including the screenshot and the PXML.xml file. The PXML.xml file is then converted to one or many .desktop files, and the desktop files, the preview.png file, and the package contents are compacted into a ZIP that then is renamed to "*.pnd"<br />
<br />
== Usage scenario ==<br />
=== Repackaging of an application from another package format ===<br />
* The user grabs the package for the application<br />
* He dumps the executable and all required libraries into a folder<br />
* He dumps a "screenshot.png" file into the folder that he's made<br />
* He copies the ".desktop" file(s) for the application from the old package, opens them, replaces "Exec=/usr/bin/bla" with "Exec=./bla" and saves them in the directory<br />
* He uses a ZIP archiver to make a ZIP out of the folder, making sure that he sets the "uncompressed" option.<br />
<br />
=== Creating PNDs as part of a build process ===<br />
* The build tool creates a directory with all of the necessary information like above and invokes the "zip" utility to compress the folder.<br />
<br />
=== Accessing a PND's contents for the... ===<br />
<br />
==== ...lazy programmer ====<br />
* The programmer extracts the PND into a folder via libzip and accesses its contents.<br />
<br />
==== ...pragmatic programmer ====<br />
* The programmer uses zipfs to mount the ZIP and accesses the file's contents.<br />
<br />
==== ...smart performance-aware programmer ====<br />
* The programmer fseek()'s the ZIP file until he finds 0x04034b50.<br />
* He jumps 18 bytes forward and reads the int at that location, storing it in "length".<br />
* He jumps 4 bytes and checks if this int matches "length". If it doesn't, it means that the file is compressed, and an error is reported.<br />
* He then jumps forward 4 bytes and stores the short at that location in a variable "nameLength".<br />
* Another 2-byte jump gets the short "extensionsLength".<br />
* He then jumps 2 bytes and reads "nameLength" amount of bytes from the file.<br />
* He then uses strcmp() to see if this string matches the sought-after file. If not, he continues the fseek().<br />
* The programmer now skips "entensionsLength" amount of bytes.<br />
* The programmer reads "length" amount of bytes from the file, and uses this as the sought-after file data.<br />
<br />
==== ...performance fascist/programmer who wants low seek times ====<br />
* The programmer fseek()'s the file from the end until he finds 0x02014b50.<br />
* He then uses the following table to get the information he needs:<br />
{|class="wikitable"<br />
|+ ZIP central directory file header<br />
|-<br />
! Offset !! Bytes !! Description<br />
|-<br />
| 0 || 4 || Central directory file header signature = 0x02014b50<br />
|-<br />
| 4 || 2 || Version made by<br />
|-<br />
| 6 || 2 || Version needed to extract (minimum)<br />
|-<br />
| 8 || 2 || General purpose bit flag<br />
|-<br />
| 10 || 2 || Compression method<br />
|-<br />
| 12 || 2 || File last modification time<br />
|-<br />
| 14 || 2 || File last modification date<br />
|-<br />
| 16 || 4 || CRC-32<br />
|-<br />
| 20 || 4 || Compressed size<br />
|-<br />
| 24 || 4 || Uncompressed size<br />
|-<br />
| 28 || 2 || File name length (''n'')<br />
|-<br />
| 30 || 2 || Extra field length (''m'')<br />
|-<br />
| 32 || 2 || File comment length (''k'')<br />
|-<br />
| 34 || 2 || Disk number where file starts<br />
|-<br />
| 36 || 2 || Internal file attributes<br />
|-<br />
| 38 || 4 || External file attributes<br />
|-<br />
| 42 || 4 || Relative offset of local file header<br />
|-<br />
| 46 || ''n'' || File name<br />
|-<br />
| 46+''n'' || ''m'' || Extra field<br />
|-<br />
| 46+''n''+''m'' || ''k'' || File comment<br />
|}<br />
<br />
* The relative file offset can then be used to jump to the file in question. The file is then scanned as described in the previous section.<br />
<br />
== The 2014/Pyra effort ==<br />
<br />
A new format is being developed for the DragonBox Pyra handheld. Here are the requirements/notes:<br />
<br />
{| class="wikitable"<br />
|-<br />
! Requirement !! Solutions !! Importance !! Status (decided, undecided, post-proned, won't fix, ...) !! Notes<br />
|-<br />
| Should the new format be supported by Pandora AND Pyra? ||<br />
* Yes<br />
** Makes releasing compatible applications easier.<br />
* no<br />
** Pandora is softfp and Pyra is hardfp - it's unlikely one package could work on both systems anyway.<br />
** For current OpenPandora, we can already run Debian on SD-card.<br />
*** Maybe we can provide Pyra apps for OpenPandora using the Debian-on-sd-card opposed to Debian-on-internal-space<br />
|| Very high || Decided: compatibility with SuperZaxxon in the future PND system is not important ||<br />
* http://boards.openpandora.org/topic/15530-the-software-side/page-7#entry310691<br />
* http://boards.openpandora.org/topic/15613-drop-support-for-superzaxxon-in-new-pnd-format/<br />
|-<br />
| API/Daemon to switch between joystick/keyboard mode || || || ||<br />
* Not exactly a PND-error<br />
* aTc gathered a nice list of notes: http://boards.openpandora.org/topic/15587-userspace-input-daemon/#entry311623<br />
|-<br />
| Guideline for consistent use of gaming buttons:<br />
* It should reduce the need of patches on top of the ported software<br />
||<br />
* A keyboard/joystick remapping function<br />
* A guideline for software developers<br />
|| || || No single-solution, but a set of them<br />
|-<br />
| Easier to parse than current PNDs and less fragile ||<br />
* header + metadata (UBJSON) + icon + screenshots + squashfs<br />
* tar archive<br />
** more future proof<br />
** easier for shell scripts<br />
*** with dd you can extract the tar part and so<br />
** filenames and file permissions are overhead<br />
*** But the overhead is too small to be significant?<br />
** easier to modify after-compiled<br />
*** not really a problem, because the user should be contributing to the recipe anyway<br />
*** a specialized tool can be create if another solution is chosen<br />
* zipfs uncompressed filesystem<br />
** See above sections<br />
* ar archive<br />
** Like .deb format<br />
|| || ||<br />
|-<br />
| Should the metadata include packages dependencies? ||<br />
* Yes<br />
** apt-get install package --print-uris will print download uris for dep packages. Using this uris download packages and extract all (dpkg -x 1.deb ./extracted)<br />
*** Not enough. It'll need two passes between an online and an offline system. It's impossible to be sure that an offline installation will work in one pass without including the whole repo: http://boards.openpandora.org/topic/15530-the-software-side/page-9#entry311602<br />
** EvilDragon voted for yes-and-no? http://boards.openpandora.org/topic/15530-the-software-side/page-5#entry310590<br />
*** Or does it mean a minimum "os-version required" metadata on the package and let the user update the system to have the latest libraries available?<br />
*** Possibly he doesn't care and just want something that just works: http://boards.openpandora.org/topic/15574-confliting-requirements/page-2#entry311580<br />
** http://boards.openpandora.org/topic/15530-the-software-side/page-5#entry310627<br />
* No.<br />
** EvilDragon commented that he wanna a plug-and-play-and-works system ( http://boards.openpandora.org/topic/15530-the-software-side/page-3#entry310412 )<br />
** If the user is lacking of internet access, "plug-and-play" package won't work<br />
** .deb for general-purpose software<br />
* Complete user-space like containment features (see http://www.docker.io/ for an example)<br />
|| || Decided ||<br />
* After a long heated discussion, general consensus seems to be:<br />
** Tune the makepnd script to create flat PNDs (packages which only deps are the minimum required set already present) and light PNDs (packages with .deb dependencies).<br />
*** Both packages would be included in the repo and user download what he likes more.<br />
|-<br />
| Minimum set of dependencies (SDL, ...) || || || WIP || Progress is happenning on github:<br />
* https://github.com/Pyra-Handheld/TODO/issues/5<br />
* https://github.com/Pyra-Handheld/TODO/issues/7<br />
|-<br />
| It should be click-and-run ||<br />
* binfmt<br />
** http://boards.openpandora.org/topic/15530-the-software-side/page-11#entry316896<br />
* Sheebang<br />
* File (extension) association<br />
|| || ||<br />
|-<br />
| Less duplication of metadata (c.f. PXML files) || || || ||<br />
|-<br />
| AUR-like system:<br />
* Building recipes is easier to replicate and contribute<br />
* If maintainer is jerk, moderator can move his maintanership<br />
|| Fork makepkg || || [http://boards.openpandora.org/topic/15530-the-software-side/page-7#entry310691 Cloudef is working on this task] ||<br />
|-<br />
| A compatibility layer for the current *.PND system:<br />
* The more transparent, the better<br />
||<br />
* Dual-boot<br />
* Virtual Machine<br />
* Tune Debian system to include all libraries used in current SuperZaxxon<br />
** OpenEmbedded it's based on the Debian, afterall<br />
** Probably won't happen because of hard/soft float point library issues<br />
* chroot<br />
** chroot doesn't start services like the real system, then the services of the chroot must be replaced/configured<br />
|| High || ||<br />
* Pyra will have larger internal space, then duplicate system isn't a large problem<br />
* The compatibility layer should have its own table?<br />
* https://github.com/Pyra-Handheld/TODO/issues/2<br />
|-<br />
| Cross-building? || || || Undecided ||<br />
|-<br />
| Home-dir of applications ||<br />
* Maybe one per SD-card<br />
* Union-fs? Aufs? [http://pdos.csail.mit.edu/mbox/ Mbox]?<br />
* overridable by env var?<br />
|| || ||<br />
|-<br />
| Overridable scripts ||<br />
* By creating files with the right name, you can override files in the package (e.g. scripts)<br />
* Yes:<br />
** Cool for hackers<br />
* No:<br />
** Necessitates a union filesystem<br />
** Not absolutely necessary - hackers can just rebuild the package.<br />
|| || ||<br />
|-<br />
| File-extension? ||<br />
* .pyra<br />
** We're not using fucking DOS for lord-sake, what's the point of use 3-letters extension?<br />
* .pyd for pyra-data<br />
* .pyr<br />
* .pp for pyra program<br />
* .pa for pyra app<br />
* .drp for dragonbox pyra<br />
** Why limit this to just the Pyra? Could be useful in future systems/elsewhere too.<br />
* ...<br />
|| || undecided. || Maybe a series of polls on the boards? ;)<br />
|-<br />
| Keep what is good || Document all of the good features and port the content to this table || High || Not started ||<br />
* http://boards.openpandora.org/topic/15613-drop-support-for-superzaxxon-in-new-pnd-format/#entry316469<br />
|-<br />
| Extra bonus points for a design that can easily be used on other distros and devices as well || || Optional || Won't fix ||<br />
* .deb dependencies already kills the system-agnostic system<br />
** Unless we submit our changes to upstream Debian, which is unlikely to happen<br />
|-<br />
| Multiple binaries/icons per app || || || Decided: it's necessary: http://boards.openpandora.org/topic/15613-drop-support-for-superzaxxon-in-new-pnd-format/page-2#entry316974 || Somebody (lost on the sea of comments) didn't like this design, but it should be possible for special software<br />
|-<br />
| Delta packages || || || || http://boards.openpandora.org/topic/15530-the-software-side/page-9#entry311600<br />
|-<br />
| PATH handling || || || || http://boards.openpandora.org/topic/15530-the-software-side/page-9#entry311618<br />
|}<br />
<br />
[[Category:PND]]<br />
[[Category:Ideas and proposals]]</div>Vinipsmakerhttps://pandorawiki.org/index.php?title=New_PND_format&diff=29540New PND format2014-02-20T21:29:22Z<p>Vinipsmaker: binfmt update</p>
<hr />
<div>The current PND format has some shortcomings as listed below. This page should serve as a discussion page/white board for how the format could be improved.<br />
<br />
''' Proposal for discussion, this is just some opening shots '''<br />
<br />
== Current situation ==<br />
The current (ISO-based) PND format has the following shortcomings:<br />
<br />
* It currently uses ISO, SquashFS, and other filesystems... This means that there's no standard to follow, and libpnd needs to carry around support for a multitude of file systems.<br />
* It (often, but not always) uses the [http://en.wikipedia.org/wiki/ISO_9660 ISO file system] which is inefficient at storing data, because:<br />
** It contains duplicate headers for each file entry[http://users.telenet.be/it3.consultants.bvba/handouts/ISO9960.html] (one version with big-endian integers, one version with little-endian integers[http://en.wikipedia.org/wiki/Endianness]) which leads to many kilobytes of wasted space.<br />
** Its file tables are fixed-size, and there are therefore limitations on how many folders you can have, what names you can give your files, how big your files can be etc. For instance, files can only have names that are max 31 characters long, all upper-case, and limited to the ASCII character encoding, and an ISO does only support a folder depth of 8[http://www.on-time.com/rtos-32-docs/rtfiles-32/programming-manual/iso-9660/]. To support more, the Joliet file system extension is needed (or isofs won't recognize all file paths), and the Joliet header can be placed practically anywhere in the file, which means additional seek times.<br />
** It needs various file system extensions to behave correctly, like for example [http://en.wikipedia.org/wiki/Rock_Ridge Rock Ridge Interchange Protocol] and [http://en.wikipedia.org/wiki/Joliet_(file_system) Joliet]. Without them, it becomes pretty useless as demonstrated above, and with them, it becomes difficult to read by a tool (You can only use isofs! There are no other tools out there for programmers to use!).<br />
** It's very difficult to use since you need special ISO making tools to create the image. ISO is a relatively rare format if you're not technically inclined.<br />
** If you want to add or remove files from it, it's near impossible if you use a compact ISO file, since there's no way that you can "expand" an existing ISO file.<br />
* The PND header data is at the end (or in the middle of the file if a screenshot is included) which makes it impossible for e.g. libmagic to recognize the PND file. It will instead recognize it as an ISO file. Having the header data at the end also makes it take a very long time to find the data, making tools and the libpnd library very inefficient.<br />
* The PND file uses a custom XML format for its metadata. There's no reason to do this, especially since the established ".desktop" file format fills exactly the same function.<br />
(note: ".desktop" does not contain all metadata we have wanted over time, but it's of course possible to add extensions à la "X-Pandora-Whatever=xyz")<br />
* There's no "index" for the PND file. The whole file has to be scanned (albeit backwards) to find a PXML file, and there's a big chance for false positives etc.<br />
* Data is just appended linearly to the file so there's no order. If the format is to be extended (to e.g. include an icon file after the screenshot file), should the data just be appended as well?<br />
* UTF-8 is strictly the only encoding that is supported. If you make your PXML on a Windows machine, it won't work.<br />
<br />
== Proposed revisions ==<br />
=== Step 1: File system ===<br />
The file system for PNDs should be replaced by the ''uncompressed'' ZIP archive format. ZIP has the advantage that it's incredibly compact, and uncompressed ZIP makes it possible to read data from the file without having to do any decompression.<br />
<br />
ZIP files are mountable using various implementations of zipfs, and most of these implementations won't store files in memory when they are being read if the ZIP file is uncompressed.<br />
<br />
=== Step 2: Metadata ===<br />
PND files should no longer require special tools for them to be created. Therefore, since ZIP files support random access on files, it shouldn't be necessary to append/prepend metadata to the file. The user simply includes the "PXML.xml" and "preview.png" files inside of the ZIP, and any tools that need information about the PND can simply go to the central directory of the ZIP file (or use a simple ZIP library to do it for them) and get the location of the file inside of the ZIP. This should also dramatically decrease loading times for PNDs.<br />
<br />
Note; performance testing is needed; at the time we started, zip was enormously slower than plain ISO; with driver changes, it may or may not be so, hence our adopted multiple-filesystem-type system. zipfs is one possible option.<br />
<br />
=== Step 3: Metadata format ===<br />
Current PND files include so-called "PXML.xml" files. These files have a custom XML format that has a strange structure.<br />
<br />
These files should be replaced by ".desktop" files. A PND can contain one or more ".desktop" files in its root directory that specify how an application should be launched. PND tools simply use all ".desktop" files they can find in the PND when creating launchers for the contents of the PND.<br />
<br />
== Advantages ==<br />
* There don't have to be ''any'' special tools for reading PND files. The package can be run on any platform using any programming language that can read ZIP files.<br />
* We can use existing facilities to manage the launching of applications. The ".desktop" files can basically be copied without modification into standard locations of the system, and all launchers will become aware of them.<br />
* Reading PND files becomes easier (because of better tool support... sorry, but you can't link to libpnd in all programming languages) and quicker (becuase of the central directory for random file access).<br />
<br />
=== Benchmarks ===<br />
A benchmark was carried through to measure the performance of various package implementations.<br />
The compared systems were:<br />
* ZIP-based uncompressed packages using fuse-zip (a zipfs implementation) to read the files<br />
* ZIP-based compressed packages also using fuse-zip.<br />
* ISO8859-based packages using isofs<br />
* CramFS-based packages.<br />
<br />
A total of 9 files were generated for the test. They have sizes 1, 2, 3, 4, 16, 32, 64, 128 and 256 MB, and were generated via "dd if=/dev/urandom of=$file bs=1M count=$size".<br />
<br />
Because the files contain random data, they did not compress very well. The resulting files were:<br />
* "zipcompressed.zip" 511.1 MB<br />
* "zipuncompressed.zip" 511.0 MB<br />
* "iso.iso" 511.3 MB<br />
* "cramfs.image" 95.4 MB (WOW!)<br />
<br />
Some notes:<br />
* No write tests were done for obvious reasons<br />
* No random access tests were done since the Pandora will use SD cards and thus not get penalized from random access, and that's not what we're interested in anyways.<br />
* Before each test, the commands "sync; echo 3 > /proc/sys/vm/drop_caches" were run.<br />
<br />
==== First test: linear read time ====<br />
Command: "time cat mountpoint/* > /dev/null"<br />
<br />
* ISO: 8.119 seconds<br />
* CramFS: 1.488 seconds (WOW!)<br />
* Zip (compressed): 8.535 seconds<br />
* Zip (uncompressed): 8.290 seconds<br />
<br />
==== Second test: RAM usage ====<br />
-- TODO, have to find a good method of measuring this --<br />
<br />
== Remaining issues ==<br />
* How should libpnd tell the difference between old and new PND files? Should it depend on the "file" tool, should it run its own recognition algorithm, or what? Should the extension be changed to e.g. ".box" (Which was proposed in a thread)?<br />
* How to make sure that the ZIP files are uncompressed? Should we provide a script that "uncompresses" an ordinary ZIP file?<br />
* Is cramfs better than ZIP?<br />
** cramfs cannot be uncompressed like ZIP can<br />
** If you compare compressed ZIP and cramfs, cramfs is more efficient.<br />
** cramfs is harder to use than ZIP (for the developer).<br />
** cramfs ''needs'' to have all of its opened files in memory.<br />
<br />
== Upgrade path from the old PND format ==<br />
A simple script can be written that extracts the old PND file, including the screenshot and the PXML.xml file. The PXML.xml file is then converted to one or many .desktop files, and the desktop files, the preview.png file, and the package contents are compacted into a ZIP that then is renamed to "*.pnd"<br />
<br />
== Usage scenario ==<br />
=== Repackaging of an application from another package format ===<br />
* The user grabs the package for the application<br />
* He dumps the executable and all required libraries into a folder<br />
* He dumps a "screenshot.png" file into the folder that he's made<br />
* He copies the ".desktop" file(s) for the application from the old package, opens them, replaces "Exec=/usr/bin/bla" with "Exec=./bla" and saves them in the directory<br />
* He uses a ZIP archiver to make a ZIP out of the folder, making sure that he sets the "uncompressed" option.<br />
<br />
=== Creating PNDs as part of a build process ===<br />
* The build tool creates a directory with all of the necessary information like above and invokes the "zip" utility to compress the folder.<br />
<br />
=== Accessing a PND's contents for the... ===<br />
<br />
==== ...lazy programmer ====<br />
* The programmer extracts the PND into a folder via libzip and accesses its contents.<br />
<br />
==== ...pragmatic programmer ====<br />
* The programmer uses zipfs to mount the ZIP and accesses the file's contents.<br />
<br />
==== ...smart performance-aware programmer ====<br />
* The programmer fseek()'s the ZIP file until he finds 0x04034b50.<br />
* He jumps 18 bytes forward and reads the int at that location, storing it in "length".<br />
* He jumps 4 bytes and checks if this int matches "length". If it doesn't, it means that the file is compressed, and an error is reported.<br />
* He then jumps forward 4 bytes and stores the short at that location in a variable "nameLength".<br />
* Another 2-byte jump gets the short "extensionsLength".<br />
* He then jumps 2 bytes and reads "nameLength" amount of bytes from the file.<br />
* He then uses strcmp() to see if this string matches the sought-after file. If not, he continues the fseek().<br />
* The programmer now skips "entensionsLength" amount of bytes.<br />
* The programmer reads "length" amount of bytes from the file, and uses this as the sought-after file data.<br />
<br />
==== ...performance fascist/programmer who wants low seek times ====<br />
* The programmer fseek()'s the file from the end until he finds 0x02014b50.<br />
* He then uses the following table to get the information he needs:<br />
{|class="wikitable"<br />
|+ ZIP central directory file header<br />
|-<br />
! Offset !! Bytes !! Description<br />
|-<br />
| 0 || 4 || Central directory file header signature = 0x02014b50<br />
|-<br />
| 4 || 2 || Version made by<br />
|-<br />
| 6 || 2 || Version needed to extract (minimum)<br />
|-<br />
| 8 || 2 || General purpose bit flag<br />
|-<br />
| 10 || 2 || Compression method<br />
|-<br />
| 12 || 2 || File last modification time<br />
|-<br />
| 14 || 2 || File last modification date<br />
|-<br />
| 16 || 4 || CRC-32<br />
|-<br />
| 20 || 4 || Compressed size<br />
|-<br />
| 24 || 4 || Uncompressed size<br />
|-<br />
| 28 || 2 || File name length (''n'')<br />
|-<br />
| 30 || 2 || Extra field length (''m'')<br />
|-<br />
| 32 || 2 || File comment length (''k'')<br />
|-<br />
| 34 || 2 || Disk number where file starts<br />
|-<br />
| 36 || 2 || Internal file attributes<br />
|-<br />
| 38 || 4 || External file attributes<br />
|-<br />
| 42 || 4 || Relative offset of local file header<br />
|-<br />
| 46 || ''n'' || File name<br />
|-<br />
| 46+''n'' || ''m'' || Extra field<br />
|-<br />
| 46+''n''+''m'' || ''k'' || File comment<br />
|}<br />
<br />
* The relative file offset can then be used to jump to the file in question. The file is then scanned as described in the previous section.<br />
<br />
== The 2014/Pyra effort ==<br />
<br />
A new format is being developed for the DragonBox Pyra handheld. Here are the requirements/notes:<br />
<br />
{| class="wikitable"<br />
|-<br />
! Requirement !! Solutions !! Importance !! Status (decided, undecided, post-proned, won't fix, ...) !! Notes<br />
|-<br />
| Should the new format be supported by Pandora AND Pyra? ||<br />
* Yes<br />
** Makes releasing compatible applications easier.<br />
* no<br />
** Pandora is softfp and Pyra is hardfp - it's unlikely one package could work on both systems anyway.<br />
** For current OpenPandora, we can already run Debian on SD-card.<br />
*** Maybe we can provide Pyra apps for OpenPandora using the Debian-on-sd-card opposed to Debian-on-internal-space<br />
|| Very high || Decided: compatibility with SuperZaxxon in the future PND system is not important ||<br />
* http://boards.openpandora.org/topic/15530-the-software-side/page-7#entry310691<br />
* http://boards.openpandora.org/topic/15613-drop-support-for-superzaxxon-in-new-pnd-format/<br />
|-<br />
| API/Daemon to switch between joystick/keyboard mode || || || ||<br />
* Not exactly a PND-error<br />
* aTc gathered a nice list of notes: http://boards.openpandora.org/topic/15587-userspace-input-daemon/#entry311623<br />
|-<br />
| Guideline for consistent use of gaming buttons:<br />
* It should reduce the need of patches on top of the ported software<br />
||<br />
* A keyboard/joystick remapping function<br />
* A guideline for software developers<br />
|| || || No single-solution, but a set of them<br />
|-<br />
| Easier to parse than current PNDs and less fragile ||<br />
* header + metadata (UBJSON) + icon + screenshots + squashfs<br />
* tar archive<br />
** more future proof<br />
** easier for shell scripts<br />
*** with dd you can extract the tar part and so<br />
** filenames and file permissions are overhead<br />
*** But the overhead is too small to be significant?<br />
** easier to modify after-compiled<br />
*** not really a problem, because the user should be contributing to the recipe anyway<br />
*** a specialized tool can be create if another solution is chosen<br />
* zipfs uncompressed filesystem<br />
** See above sections<br />
* ar archive<br />
** Like .deb format<br />
|| || ||<br />
|-<br />
| Should the metadata include packages dependencies? ||<br />
* Yes<br />
** apt-get install package --print-uris will print download uris for dep packages. Using this uris download packages and extract all (dpkg -x 1.deb ./extracted)<br />
*** Not enough. It'll need two passes between an online and an offline system. It's impossible to be sure that an offline installation will work in one pass without including the whole repo: http://boards.openpandora.org/topic/15530-the-software-side/page-9#entry311602<br />
** EvilDragon voted for yes-and-no? http://boards.openpandora.org/topic/15530-the-software-side/page-5#entry310590<br />
*** Or does it mean a minimum "os-version required" metadata on the package and let the user update the system to have the latest libraries available?<br />
*** Possibly he doesn't care and just want something that just works: http://boards.openpandora.org/topic/15574-confliting-requirements/page-2#entry311580<br />
** http://boards.openpandora.org/topic/15530-the-software-side/page-5#entry310627<br />
* No.<br />
** EvilDragon commented that he wanna a plug-and-play-and-works system ( http://boards.openpandora.org/topic/15530-the-software-side/page-3#entry310412 )<br />
** If the user is lacking of internet access, "plug-and-play" package won't work<br />
** .deb for general-purpose software<br />
* Complete user-space like containment features (see http://www.docker.io/ for an example)<br />
|| || Decided ||<br />
* After a long heated discussion, general consensus seems to be:<br />
** Tune the makepnd script to create flat PNDs (packages which only deps are the minimum required set already present) and light PNDs (packages with .deb dependencies).<br />
*** Both packages would be included in the repo and user download what he likes more.<br />
|-<br />
| Minimum set of dependencies (SDL, ...) || || || WIP || Progress is happenning on github:<br />
* https://github.com/Pyra-Handheld/TODO/issues/5<br />
* https://github.com/Pyra-Handheld/TODO/issues/7<br />
|-<br />
| It should be click-and-run ||<br />
* binfmt<br />
** http://boards.openpandora.org/topic/15530-the-software-side/page-11#entry316896<br />
* Sheebang<br />
* File (extension) association<br />
|| || ||<br />
|-<br />
| Less duplication of metadata (c.f. PXML files) || || || ||<br />
|-<br />
| AUR-like system:<br />
* Building recipes is easier to replicate and contribute<br />
* If maintainer is jerk, moderator can move his maintanership<br />
|| Fork makepkg || || [http://boards.openpandora.org/topic/15530-the-software-side/page-7#entry310691 Cloudef is working on this task] ||<br />
|-<br />
| A compatibility layer for the current *.PND system:<br />
* The more transparent, the better<br />
||<br />
* Dual-boot<br />
* Virtual Machine<br />
* Tune Debian system to include all libraries used in current SuperZaxxon<br />
** OpenEmbedded it's based on the Debian, afterall<br />
** Probably won't happen because of hard/soft float point library issues<br />
* chroot<br />
** chroot doesn't start services like the real system, then the services of the chroot must be replaced/configured<br />
|| High || ||<br />
* Pyra will have larger internal space, then duplicate system isn't a large problem<br />
* The compatibility layer should have its own table?<br />
* https://github.com/Pyra-Handheld/TODO/issues/2<br />
|-<br />
| Cross-building? || || || Undecided ||<br />
|-<br />
| Home-dir of applications ||<br />
* Maybe one per SD-card<br />
* Union-fs? Aufs? [http://pdos.csail.mit.edu/mbox/ Mbox]?<br />
* overridable by env var?<br />
|| || ||<br />
|-<br />
| Overridable scripts ||<br />
* By creating files with the right name, you can override files in the package (e.g. scripts)<br />
* Yes:<br />
** Cool for hackers<br />
* No:<br />
** Necessitates a union filesystem<br />
** Not absolutely necessary - hackers can just rebuild the package.<br />
|| || ||<br />
|-<br />
| File-extension? ||<br />
* .pyra<br />
** We're not using fucking DOS for lord-sake, what's the point of use 3-letters extension?<br />
* .pyd for pyra-data<br />
* .pyr<br />
* .pp for pyra program<br />
* .pa for pyra app<br />
* .drp for dragonbox pyra<br />
** Why limit this to just the Pyra? Could be useful in future systems/elsewhere too.<br />
* ...<br />
|| || undecided. || Maybe a series of polls on the boards? ;)<br />
|-<br />
| Keep what is good || Document all of the good features and port the content to this table || High || Not started ||<br />
* http://boards.openpandora.org/topic/15613-drop-support-for-superzaxxon-in-new-pnd-format/#entry316469<br />
|-<br />
| Extra bonus points for a design that can easily be used on other distros and devices as well || || Optional || Won't fix ||<br />
* .deb dependencies already kills the system-agnostic system<br />
** Unless we submit our changes to upstream Debian, which is unlikely to happen<br />
|-<br />
| Multiple binaries/icons per app || || || || Somebody (lost on the sea of comments) didn't like this design, but it should be possible for special software<br />
|-<br />
| Delta packages || || || || http://boards.openpandora.org/topic/15530-the-software-side/page-9#entry311600<br />
|-<br />
| PATH handling || || || || http://boards.openpandora.org/topic/15530-the-software-side/page-9#entry311618<br />
|}<br />
<br />
[[Category:PND]]<br />
[[Category:Ideas and proposals]]</div>Vinipsmakerhttps://pandorawiki.org/index.php?title=New_PND_format&diff=29539New PND format2014-02-19T02:54:27Z<p>Vinipsmaker: </p>
<hr />
<div>The current PND format has some shortcomings as listed below. This page should serve as a discussion page/white board for how the format could be improved.<br />
<br />
''' Proposal for discussion, this is just some opening shots '''<br />
<br />
== Current situation ==<br />
The current (ISO-based) PND format has the following shortcomings:<br />
<br />
* It currently uses ISO, SquashFS, and other filesystems... This means that there's no standard to follow, and libpnd needs to carry around support for a multitude of file systems.<br />
* It (often, but not always) uses the [http://en.wikipedia.org/wiki/ISO_9660 ISO file system] which is inefficient at storing data, because:<br />
** It contains duplicate headers for each file entry[http://users.telenet.be/it3.consultants.bvba/handouts/ISO9960.html] (one version with big-endian integers, one version with little-endian integers[http://en.wikipedia.org/wiki/Endianness]) which leads to many kilobytes of wasted space.<br />
** Its file tables are fixed-size, and there are therefore limitations on how many folders you can have, what names you can give your files, how big your files can be etc. For instance, files can only have names that are max 31 characters long, all upper-case, and limited to the ASCII character encoding, and an ISO does only support a folder depth of 8[http://www.on-time.com/rtos-32-docs/rtfiles-32/programming-manual/iso-9660/]. To support more, the Joliet file system extension is needed (or isofs won't recognize all file paths), and the Joliet header can be placed practically anywhere in the file, which means additional seek times.<br />
** It needs various file system extensions to behave correctly, like for example [http://en.wikipedia.org/wiki/Rock_Ridge Rock Ridge Interchange Protocol] and [http://en.wikipedia.org/wiki/Joliet_(file_system) Joliet]. Without them, it becomes pretty useless as demonstrated above, and with them, it becomes difficult to read by a tool (You can only use isofs! There are no other tools out there for programmers to use!).<br />
** It's very difficult to use since you need special ISO making tools to create the image. ISO is a relatively rare format if you're not technically inclined.<br />
** If you want to add or remove files from it, it's near impossible if you use a compact ISO file, since there's no way that you can "expand" an existing ISO file.<br />
* The PND header data is at the end (or in the middle of the file if a screenshot is included) which makes it impossible for e.g. libmagic to recognize the PND file. It will instead recognize it as an ISO file. Having the header data at the end also makes it take a very long time to find the data, making tools and the libpnd library very inefficient.<br />
* The PND file uses a custom XML format for its metadata. There's no reason to do this, especially since the established ".desktop" file format fills exactly the same function.<br />
(note: ".desktop" does not contain all metadata we have wanted over time, but it's of course possible to add extensions à la "X-Pandora-Whatever=xyz")<br />
* There's no "index" for the PND file. The whole file has to be scanned (albeit backwards) to find a PXML file, and there's a big chance for false positives etc.<br />
* Data is just appended linearly to the file so there's no order. If the format is to be extended (to e.g. include an icon file after the screenshot file), should the data just be appended as well?<br />
* UTF-8 is strictly the only encoding that is supported. If you make your PXML on a Windows machine, it won't work.<br />
<br />
== Proposed revisions ==<br />
=== Step 1: File system ===<br />
The file system for PNDs should be replaced by the ''uncompressed'' ZIP archive format. ZIP has the advantage that it's incredibly compact, and uncompressed ZIP makes it possible to read data from the file without having to do any decompression.<br />
<br />
ZIP files are mountable using various implementations of zipfs, and most of these implementations won't store files in memory when they are being read if the ZIP file is uncompressed.<br />
<br />
=== Step 2: Metadata ===<br />
PND files should no longer require special tools for them to be created. Therefore, since ZIP files support random access on files, it shouldn't be necessary to append/prepend metadata to the file. The user simply includes the "PXML.xml" and "preview.png" files inside of the ZIP, and any tools that need information about the PND can simply go to the central directory of the ZIP file (or use a simple ZIP library to do it for them) and get the location of the file inside of the ZIP. This should also dramatically decrease loading times for PNDs.<br />
<br />
Note; performance testing is needed; at the time we started, zip was enormously slower than plain ISO; with driver changes, it may or may not be so, hence our adopted multiple-filesystem-type system. zipfs is one possible option.<br />
<br />
=== Step 3: Metadata format ===<br />
Current PND files include so-called "PXML.xml" files. These files have a custom XML format that has a strange structure.<br />
<br />
These files should be replaced by ".desktop" files. A PND can contain one or more ".desktop" files in its root directory that specify how an application should be launched. PND tools simply use all ".desktop" files they can find in the PND when creating launchers for the contents of the PND.<br />
<br />
== Advantages ==<br />
* There don't have to be ''any'' special tools for reading PND files. The package can be run on any platform using any programming language that can read ZIP files.<br />
* We can use existing facilities to manage the launching of applications. The ".desktop" files can basically be copied without modification into standard locations of the system, and all launchers will become aware of them.<br />
* Reading PND files becomes easier (because of better tool support... sorry, but you can't link to libpnd in all programming languages) and quicker (becuase of the central directory for random file access).<br />
<br />
=== Benchmarks ===<br />
A benchmark was carried through to measure the performance of various package implementations.<br />
The compared systems were:<br />
* ZIP-based uncompressed packages using fuse-zip (a zipfs implementation) to read the files<br />
* ZIP-based compressed packages also using fuse-zip.<br />
* ISO8859-based packages using isofs<br />
* CramFS-based packages.<br />
<br />
A total of 9 files were generated for the test. They have sizes 1, 2, 3, 4, 16, 32, 64, 128 and 256 MB, and were generated via "dd if=/dev/urandom of=$file bs=1M count=$size".<br />
<br />
Because the files contain random data, they did not compress very well. The resulting files were:<br />
* "zipcompressed.zip" 511.1 MB<br />
* "zipuncompressed.zip" 511.0 MB<br />
* "iso.iso" 511.3 MB<br />
* "cramfs.image" 95.4 MB (WOW!)<br />
<br />
Some notes:<br />
* No write tests were done for obvious reasons<br />
* No random access tests were done since the Pandora will use SD cards and thus not get penalized from random access, and that's not what we're interested in anyways.<br />
* Before each test, the commands "sync; echo 3 > /proc/sys/vm/drop_caches" were run.<br />
<br />
==== First test: linear read time ====<br />
Command: "time cat mountpoint/* > /dev/null"<br />
<br />
* ISO: 8.119 seconds<br />
* CramFS: 1.488 seconds (WOW!)<br />
* Zip (compressed): 8.535 seconds<br />
* Zip (uncompressed): 8.290 seconds<br />
<br />
==== Second test: RAM usage ====<br />
-- TODO, have to find a good method of measuring this --<br />
<br />
== Remaining issues ==<br />
* How should libpnd tell the difference between old and new PND files? Should it depend on the "file" tool, should it run its own recognition algorithm, or what? Should the extension be changed to e.g. ".box" (Which was proposed in a thread)?<br />
* How to make sure that the ZIP files are uncompressed? Should we provide a script that "uncompresses" an ordinary ZIP file?<br />
* Is cramfs better than ZIP?<br />
** cramfs cannot be uncompressed like ZIP can<br />
** If you compare compressed ZIP and cramfs, cramfs is more efficient.<br />
** cramfs is harder to use than ZIP (for the developer).<br />
** cramfs ''needs'' to have all of its opened files in memory.<br />
<br />
== Upgrade path from the old PND format ==<br />
A simple script can be written that extracts the old PND file, including the screenshot and the PXML.xml file. The PXML.xml file is then converted to one or many .desktop files, and the desktop files, the preview.png file, and the package contents are compacted into a ZIP that then is renamed to "*.pnd"<br />
<br />
== Usage scenario ==<br />
=== Repackaging of an application from another package format ===<br />
* The user grabs the package for the application<br />
* He dumps the executable and all required libraries into a folder<br />
* He dumps a "screenshot.png" file into the folder that he's made<br />
* He copies the ".desktop" file(s) for the application from the old package, opens them, replaces "Exec=/usr/bin/bla" with "Exec=./bla" and saves them in the directory<br />
* He uses a ZIP archiver to make a ZIP out of the folder, making sure that he sets the "uncompressed" option.<br />
<br />
=== Creating PNDs as part of a build process ===<br />
* The build tool creates a directory with all of the necessary information like above and invokes the "zip" utility to compress the folder.<br />
<br />
=== Accessing a PND's contents for the... ===<br />
<br />
==== ...lazy programmer ====<br />
* The programmer extracts the PND into a folder via libzip and accesses its contents.<br />
<br />
==== ...pragmatic programmer ====<br />
* The programmer uses zipfs to mount the ZIP and accesses the file's contents.<br />
<br />
==== ...smart performance-aware programmer ====<br />
* The programmer fseek()'s the ZIP file until he finds 0x04034b50.<br />
* He jumps 18 bytes forward and reads the int at that location, storing it in "length".<br />
* He jumps 4 bytes and checks if this int matches "length". If it doesn't, it means that the file is compressed, and an error is reported.<br />
* He then jumps forward 4 bytes and stores the short at that location in a variable "nameLength".<br />
* Another 2-byte jump gets the short "extensionsLength".<br />
* He then jumps 2 bytes and reads "nameLength" amount of bytes from the file.<br />
* He then uses strcmp() to see if this string matches the sought-after file. If not, he continues the fseek().<br />
* The programmer now skips "entensionsLength" amount of bytes.<br />
* The programmer reads "length" amount of bytes from the file, and uses this as the sought-after file data.<br />
<br />
==== ...performance fascist/programmer who wants low seek times ====<br />
* The programmer fseek()'s the file from the end until he finds 0x02014b50.<br />
* He then uses the following table to get the information he needs:<br />
{|class="wikitable"<br />
|+ ZIP central directory file header<br />
|-<br />
! Offset !! Bytes !! Description<br />
|-<br />
| 0 || 4 || Central directory file header signature = 0x02014b50<br />
|-<br />
| 4 || 2 || Version made by<br />
|-<br />
| 6 || 2 || Version needed to extract (minimum)<br />
|-<br />
| 8 || 2 || General purpose bit flag<br />
|-<br />
| 10 || 2 || Compression method<br />
|-<br />
| 12 || 2 || File last modification time<br />
|-<br />
| 14 || 2 || File last modification date<br />
|-<br />
| 16 || 4 || CRC-32<br />
|-<br />
| 20 || 4 || Compressed size<br />
|-<br />
| 24 || 4 || Uncompressed size<br />
|-<br />
| 28 || 2 || File name length (''n'')<br />
|-<br />
| 30 || 2 || Extra field length (''m'')<br />
|-<br />
| 32 || 2 || File comment length (''k'')<br />
|-<br />
| 34 || 2 || Disk number where file starts<br />
|-<br />
| 36 || 2 || Internal file attributes<br />
|-<br />
| 38 || 4 || External file attributes<br />
|-<br />
| 42 || 4 || Relative offset of local file header<br />
|-<br />
| 46 || ''n'' || File name<br />
|-<br />
| 46+''n'' || ''m'' || Extra field<br />
|-<br />
| 46+''n''+''m'' || ''k'' || File comment<br />
|}<br />
<br />
* The relative file offset can then be used to jump to the file in question. The file is then scanned as described in the previous section.<br />
<br />
== The 2014/Pyra effort ==<br />
<br />
A new format is being developed for the DragonBox Pyra handheld. Here are the requirements/notes:<br />
<br />
{| class="wikitable"<br />
|-<br />
! Requirement !! Solutions !! Importance !! Status (decided, undecided, post-proned, won't fix, ...) !! Notes<br />
|-<br />
| Should the new format be supported by Pandora AND Pyra? ||<br />
* Yes<br />
** Makes releasing compatible applications easier.<br />
* no<br />
** Pandora is softfp and Pyra is hardfp - it's unlikely one package could work on both systems anyway.<br />
** For current OpenPandora, we can already run Debian on SD-card.<br />
*** Maybe we can provide Pyra apps for OpenPandora using the Debian-on-sd-card opposed to Debian-on-internal-space<br />
|| Very high || Decided: compatibility with SuperZaxxon in the future PND system is not important ||<br />
* http://boards.openpandora.org/topic/15530-the-software-side/page-7#entry310691<br />
* http://boards.openpandora.org/topic/15613-drop-support-for-superzaxxon-in-new-pnd-format/<br />
|-<br />
| API/Daemon to switch between joystick/keyboard mode || || || ||<br />
* Not exactly a PND-error<br />
* aTc gathered a nice list of notes: http://boards.openpandora.org/topic/15587-userspace-input-daemon/#entry311623<br />
|-<br />
| Guideline for consistent use of gaming buttons:<br />
* It should reduce the need of patches on top of the ported software<br />
||<br />
* A keyboard/joystick remapping function<br />
* A guideline for software developers<br />
|| || || No single-solution, but a set of them<br />
|-<br />
| Easier to parse than current PNDs and less fragile ||<br />
* header + metadata (UBJSON) + icon + screenshots + squashfs<br />
* tar archive<br />
** more future proof<br />
** easier for shell scripts<br />
*** with dd you can extract the tar part and so<br />
** filenames and file permissions are overhead<br />
*** But the overhead is too small to be significant?<br />
** easier to modify after-compiled<br />
*** not really a problem, because the user should be contributing to the recipe anyway<br />
*** a specialized tool can be create if another solution is chosen<br />
* zipfs uncompressed filesystem<br />
** See above sections<br />
* ar archive<br />
** Like .deb format<br />
|| || ||<br />
|-<br />
| Should the metadata include packages dependencies? ||<br />
* Yes<br />
** apt-get install package --print-uris will print download uris for dep packages. Using this uris download packages and extract all (dpkg -x 1.deb ./extracted)<br />
*** Not enough. It'll need two passes between an online and an offline system. It's impossible to be sure that an offline installation will work in one pass without including the whole repo: http://boards.openpandora.org/topic/15530-the-software-side/page-9#entry311602<br />
** EvilDragon voted for yes-and-no? http://boards.openpandora.org/topic/15530-the-software-side/page-5#entry310590<br />
*** Or does it mean a minimum "os-version required" metadata on the package and let the user update the system to have the latest libraries available?<br />
*** Possibly he doesn't care and just want something that just works: http://boards.openpandora.org/topic/15574-confliting-requirements/page-2#entry311580<br />
** http://boards.openpandora.org/topic/15530-the-software-side/page-5#entry310627<br />
* No.<br />
** EvilDragon commented that he wanna a plug-and-play-and-works system ( http://boards.openpandora.org/topic/15530-the-software-side/page-3#entry310412 )<br />
** If the user is lacking of internet access, "plug-and-play" package won't work<br />
** .deb for general-purpose software<br />
* Complete user-space like containment features (see http://www.docker.io/ for an example)<br />
|| || Decided ||<br />
* After a long heated discussion, general consensus seems to be:<br />
** Tune the makepnd script to create flat PNDs (packages which only deps are the minimum required set already present) and light PNDs (packages with .deb dependencies).<br />
*** Both packages would be included in the repo and user download what he likes more.<br />
|-<br />
| Minimum set of dependencies (SDL, ...) || || || WIP || Progress is happenning on github:<br />
* https://github.com/Pyra-Handheld/TODO/issues/5<br />
* https://github.com/Pyra-Handheld/TODO/issues/7<br />
|-<br />
| It should be click-and-run ||<br />
* binfmt<br />
* Sheebang<br />
* File (extension) association<br />
|| || ||<br />
|-<br />
| Less duplication of metadata (c.f. PXML files) || || || ||<br />
|-<br />
| AUR-like system:<br />
* Building recipes is easier to replicate and contribute<br />
* If maintainer is jerk, moderator can move his maintanership<br />
|| Fork makepkg || || [http://boards.openpandora.org/topic/15530-the-software-side/page-7#entry310691 Cloudef is working on this task] ||<br />
|-<br />
| A compatibility layer for the current *.PND system:<br />
* The more transparent, the better<br />
||<br />
* Dual-boot<br />
* Virtual Machine<br />
* Tune Debian system to include all libraries used in current SuperZaxxon<br />
** OpenEmbedded it's based on the Debian, afterall<br />
** Probably won't happen because of hard/soft float point library issues<br />
* chroot<br />
** chroot doesn't start services like the real system, then the services of the chroot must be replaced/configured<br />
|| High || ||<br />
* Pyra will have larger internal space, then duplicate system isn't a large problem<br />
* The compatibility layer should have its own table?<br />
* https://github.com/Pyra-Handheld/TODO/issues/2<br />
|-<br />
| Cross-building? || || || Undecided ||<br />
|-<br />
| Home-dir of applications ||<br />
* Maybe one per SD-card<br />
* Union-fs? Aufs? [http://pdos.csail.mit.edu/mbox/ Mbox]?<br />
* overridable by env var?<br />
|| || ||<br />
|-<br />
| Overridable scripts ||<br />
* By creating files with the right name, you can override files in the package (e.g. scripts)<br />
* Yes:<br />
** Cool for hackers<br />
* No:<br />
** Necessitates a union filesystem<br />
** Not absolutely necessary - hackers can just rebuild the package.<br />
|| || ||<br />
|-<br />
| File-extension? ||<br />
* .pyra<br />
** We're not using fucking DOS for lord-sake, what's the point of use 3-letters extension?<br />
* .pyd for pyra-data<br />
* .pyr<br />
* .pp for pyra program<br />
* .pa for pyra app<br />
* .drp for dragonbox pyra<br />
** Why limit this to just the Pyra? Could be useful in future systems/elsewhere too.<br />
* ...<br />
|| || undecided. || Maybe a series of polls on the boards? ;)<br />
|-<br />
| Keep what is good || Document all of the good features and port the content to this table || High || Not started ||<br />
* http://boards.openpandora.org/topic/15613-drop-support-for-superzaxxon-in-new-pnd-format/#entry316469<br />
|-<br />
| Extra bonus points for a design that can easily be used on other distros and devices as well || || Optional || Won't fix ||<br />
* .deb dependencies already kills the system-agnostic system<br />
** Unless we submit our changes to upstream Debian, which is unlikely to happen<br />
|-<br />
| Multiple binaries/icons per app || || || || Somebody (lost on the sea of comments) didn't like this design, but it should be possible for special software<br />
|-<br />
| Delta packages || || || || http://boards.openpandora.org/topic/15530-the-software-side/page-9#entry311600<br />
|-<br />
| PATH handling || || || || http://boards.openpandora.org/topic/15530-the-software-side/page-9#entry311618<br />
|}<br />
<br />
[[Category:PND]]<br />
[[Category:Ideas and proposals]]</div>Vinipsmakerhttps://pandorawiki.org/index.php?title=Pandian&diff=29519Pandian2014-02-13T07:05:35Z<p>Vinipsmaker: /* Intuitive Openbox menu template */</p>
<hr />
<div>[[File:Post-3032-0-90501200-1380479048.png|thumbnail|Logo]]<br />
<br />
'''Pandian is an debian-based armhf desktop and CLI distro for your Pandora.'''<br />
Ideas, Bugs, or feature-requests? Report to:<br />
http://www.evilbrain.de/projects/pandian/issues<br />
<br />
=== Main Features ===<br />
* The debian way.<br />
* Access to all packages from Debian software repositories.<br />
* Pandian hosts its own Debian repo http://pandian.openpandora.org with up to date scripts, all settings and extra programs.<br />
* armhf-build, which means '''f'''loat operations are relegated to special '''h'''ardware rather than taxing the main cpu with them (as with softfloat)<br />
<br />
== '''Pandian''' - '''MARK 2''' The newest release ==<br />
Second version of Pandian<br /><br />
'''Full Debian GNU/Linux based on sid branch with LXDE desktop environment (normal) or CLI (minimal-) build.''' <br /> <br />
'''hardfloat armv7l''' yule release '''Date: 29.12.2013'''<br />
[http://boards.openpandora.org/topic/15302-pandian-mark2/ forum link]<br />
<br />
=== '''Installation''' requirements===<br />
* '''Pandora''' ;) All versions fine?<br />
* '''SD card''' minimum 2GB ( was tested with 4GB )<br />
* '''PC''' Linux|GNU, BSD, Mac, or Windows for putting the install on the aforementioned SD-card.<br />
<br />
==== '''Download''' a version of Pandian ====<br />
There are two different versions of the "MARK2" release:<br />
{| class="wikitable"<br />
|-<br />
! Version !! Description !! Includes !! Link !! Size !! Checksums (sha256)<br />
|-<br />
| '''base''' || pandian for advanced users, without any Desktop environment but with basic network-tools ( iw, ceni ) || ceni ( network-manager) network-manager MediaPlayer ( mpd ) iw (wireless-tools) || [http://www.filedropper.com/pandian-mark2-base-hf2014-01-05 filedropper: pandian-mark2-base-hf2014-01-05.7z]<br />[http://pandian.openpandora.org/releases/pandian-mark2-base-hf_2014-01-05.7z pandian.openpandora.org: pandian-mark2-base-hf_2014-01-05.7z] || 161MB || pandian-mark2-base-hf_2014-01-05.'''7z'''↓ b11ffe9cc4fd6111775d0b48c133b1e1afcf6865de784b9865db0cea29674d85<br />pandian-mark2-base-hf_2014-01-05.'''img'''↓ c1903423a5b5e0d04563f14526e8161fbe77a102e64d5d571bcf3b87870fa211 <br />
|-<br />
| '''normal''' || pandian for regular users, with full Desktop environment (LXDE) || network-manager MediaPlayer ( mpd with gurlp ) nub-support calibrator-application medit Text editor. mpd audio player backend, glurp as frontend || [http://www.filedropper.com/pandian-mark2-hf2014-01-02 filedropper: pandian-mark2-hf2014-01-02]<br />[http://pandian.openpandora.org/releases/pandian-mark2-hf_2014-01-02.7z pandian.openpandora.org: pandian-mark2-hf_2014-01-02.7z] || 394MB || pandian-mark2-hf_2014-01-02.'''7z'''↓ 3731d70a4d8ec59e79a046f34f69c5c809be73e38cfc91a5b580478b8f8c222f<br />pandian-mark2-hf_2014-01-02.'''img'''↓ 83b8d3fa5bd0588cfb8b11cfe6b827819e6311e98396a6e32c46c9692f072e96<br />
|}<br />
Notes:<br />
* It is trivial to change from LXDE to [[Xfce]] (or any other desktop environment), adding a new user interface atop the minimal install is more advanced as manual setup of USB-Networking or wifi is needed to do so.<br />
* All images have automount ( for USB-Sticks, external Hard-Drives etc. ) and ALSA sound support<br />
<br />
==== '''Verify''' that your download is OK ====<br />
* After you download the version you picked, verify its checksum to make sure the integrity of the downloaded file matches (in being an exact copy the one on the server)<br />
<br />
#open up a terminal<br />
# "'''sha256sum ~/downloads/pandian/<image.7Z>'''" (input where you have the file and the one you chose to download)<br />
# make sure it matches the given md5sum, if not, try to download another copy. This step isnt mandatory, but its there to make sure everything is fine before we proceed.<br />
<br />
==== '''Unzip''' the downloaded image ====<br />
*Unzipping 7z archives<br />
# Open up a terminal (or do it your way)<br />
# Enter"'''7zr e <image.7z>'''" <br />
# You now have an unzipped image<br />
<br />
==== (Optional) '''Preparation''', find out where to install ====<br />
<br />
This step helps you find out where the SD card is mounted.<br /><br />
<br />
If you have an build in SD-Card-Reader this is normally /dev/mmcblk0p1 ( in Debian )<br />
If you have an USB-Card-Reader this is something like /dev/sdc ( in Debian )<br />
* Open an Terminal and input "'''lsblk'''"<br />
* You will get a list with the actual Block-Devices<br />
* The SD-Card should be of type 'disk'<br />
<br />
You can also try gparted to see which devices you have and identify the SD-card from there<br />
<br />
==== '''Install''' the image to the SD-Card ====<br />
{{warning<br />
| Warning, this will destroy everything on the device ! So please be very sure which device is you SD-Card !<br />
}}<br />
# Here it will be /dev/sd'''x''' <br />
<br />
(note that the x after the sd is the drive letter, it could be ''sd'''b''''' sd'''c''''' or ''sd'''d''''' and so on, lets say its sd'''c'''. Which means c is the drive letter for the SD-card and that sd- a and -b are in use for other things. <br /><br />
<br /><br />
When writing to the device, make sure not to include the following number, which is the partition ''sdc'''1''''' ''sdc'''2''''' etc. since we are using the base card here) <br /><br />
<br /><br />
This means considerations for partitions and filesystem are taken care of, so no need to worry about that. Do ''not'' pick the wrong drive however, since dd will happily write the image to somewhere it isn't supposed to if you just give it root privileges and hit enter.)<br /><br />
<br /><br />
#Copy image to SD-Card with "'''dd if=<image.img> of=/dev/sdx'''"<br />
<br />
You now have Pandian installed on your SD card and its ready to go. :) However after copying the image to the SD-Card resizing the partitionspartitions is needed to fully utilize the space, since by default the install is limited to 2GB. (If your card is bigger Pandian is still limited to those 2GB)<br />
<br />
====(Optional) '''Resize''' SD-Card ====<br />
<br />
* This can be done with gparted on linux|gnu<br />
* start gparted<br />
<br />
[[File:Gparted resize 001.png|800px|thumbnail|none|The actual partitions of the SD-CARD. Please select the correct device at the top right corner.]]<br />
[[File:Gparted resize 002.png|800px|thumbnail|none|Rightclick to the second partition and select Resize/Move]]<br />
[[File:Gparted resize 003.png|400px|thumbnail|none|You see the actual Size of this partition]]<br />
[[File:Gparted resize 004.png|400px|thumbnail|none|Move the slider to the right, or set "Free space following" to 0]]<br />
[[File:Gparted resize 005.png|800px|thumbnail|none|After you press the "Resize/Move" button, you get the preview of the new partition structure]]<br />
[[File:Gparted resize 006.png|800px|thumbnail|none|When you press "Apply all operations", youve got a warning that you can loss your Data.<br />THIS IS THE LAST CHANGE TO CHECK IF YOU SELECTED THE CORRECT DEVICE. <br />
<br />
<br />Press Apply if everything is correct.]]<br />
<br />
* After everything is fine, you have a fresh install of Pandian on you SD-Card<br />
<br /><br />
<br />
=== '''Startup''' wizard ===<br />
* Upon first boot of Pandian, there is a first-start-wizard which guides you through some basic settings.<br />
These are: Locale, timezone, time and date, password for admin and user setup and finally calibration of the touchscreen.<br />
The script is located in /usr/sbin/pdSetup should you need to run it again.<br />
<br />
==== Navigation ====<br />
* Inside a dialog you can navigate with the cross/dpad<br />
* Hit space to select a checkbox<br />
* Moving between <Ok> and <Cancel> can be down with the use of TAB ( Tap Fn then tap Space )<br />
* Acknowledge with Enter<br />
<br />
*All done, enjoy :)<br />
<br />
[[File:PandianDesktop.png|frameless]]<br />
<br />
=== '''Network''' connection instructions:===<br />
<br />
====normal ====<br />
* Use network-manager<br />
<br />
====minimal====<br />
* Use ceni network manager<br />
<br />
===(Optional) '''Update''' of Pandian system ===<br />
* After you setup your Pandora, it is useful to update your system. It is a rolling release, which means since the image was made, the system is forever updated with new packages.<br />
* open a terminal<br />
* get root ( command is: "su" )<br />
* enter "apt-get update" to download package information from the repositories<br />
* enter "apt-get upgrade" to update all packages that are out of date<br />
(pandian packages are also updated this way :)<br />
* Note: to update only the pandian-packages enter "apt-get upgrade pandian*"<br />
<br />
<br />
===== Upgrade from "minimal" to "normal" =====<br />
It is possible to upgrade from the "minimal"-image to the "normal"-image.<br />
This can simlpy be done by installing the pandian-lxde debian package<br />
* get an connection to the internet(tm) with ceni<br />
* open an terminal<br />
* get root ( with command: "su" )<br />
* update package list with: "apt-get update"<br />
* install pandian-lxde with "apt-get install pandian-lxde"<br />
* after everything is finished login with the user you setup before<br />
* open an terminal<br />
* get root ( with command: "su" )<br />
* load keymap with "xmodmap /etc/skel/.pndXmodmap"<br />
* the easiest way is to activate the first-start-wizard with "insserv firstuse"<br />
* reboot and go through the wizard<br />
<br />
===(Optional) '''Information''' ===<br />
<br />
=== Additional Users/Groups ===<br />
There are additional Users/Groups added to the standard debian-users<br />
==== Groups ====<br />
* admin<br />
:This superuser privilege is used by sudo for admin task like shutdown etc.<br />
* mpd<br />
:This user/group has access to mpd-related files ( socket, pid-files )<br />
<br />
==== Power-Modes ====<br />
===== LID-Close =====<br />
* If the lid is closed only the screen will be disabled<br />
===== Low-Power-Mode / Shutdown =====<br />
[[File:Lead Photo For Power modes0-2920264867079787.jpg|thumbnail|right|Switch]]<br />
* Switch to left: Pandora will go to low power mode ( reduce CPU, Disable LID, Pause processes )<br />
* Switch to right: Shutdown the pandora<br />
<br />
==== pdcmd ====<br />
* This script handles all system-related stuff, like lid, low-power, usb or wifi.<br />
* This should run as root<br />
<br /><br />
Usage:<br />
{| class="wikitable"<br />
|-<br />
! Command !! What's done<br />
|-<br />
| /usr/sbin/pdcmd LCDBright + || Increase the light of the lid<br />
|-<br />
| /usr/sbin/pdcmd LCDBright - || Decrease the light of the lid<br />
|-<br />
| /usr/sbin/pdcmd LCDBright 10 || Set the Brightness of the lid to 10<br />
|-<br />
| /usr/sbin/pdcmd LCDTrigger || Trigger the lid ( WARNING: THIS DISABLES THE LID ! )<br />
|-<br />
| /usr/sbin/pdcmd USBNetTrigger || Enable the USB-Networking (module g_ether). If usbnetworking is on, it will disable it<br />
|-<br />
| /usr/sbin/pdcmd USBHostTrigger || Enable the USB-Host (module ehci-hcd). If USB-Host is on, it will disable it<br />
|-<br />
| /usr/sbin/pdcmd WLANTrigger || Enable Wifi. If Wifi is enabled it will be disabled<br />
|}<br />
<br />
==== Several tweaks ====<br />
===== Making the Pandora Button work like the Windows key =====<br />
Many users complained, their Pandora Button has no function in Pandian. This can be changed by mapping the button as ''"mod4"''. <br />
*Make sure, the file .pndXmodmap is read in on startup. Open the file ''.xsessionrc'' in your ''home'' directory and, if not existent, add following to the last line:<br />
xmodmap /home/yourUsername/.pndXmodmap<br />
*There must be the absolute path (no "~/").<br />
*Insert in the last line of the .pndXmodmap from your ''home'' directory:<br />
add mod4 = XF86MenuKB<br />
<br />
===== Openbox =====<br />
Pandian comes with an unconfigured Openbox as a possible session. Here are some config templates.<br />
<br />
====== Use the Pandora key to start the menu ======<br />
*If not existent,<br />
cp /etc/xdg/openbox/rc.xml /home/yourUserName/.config/openbox/<br />
* Open the rc.xml with a text editor and search for the section <keyboard>.<br />
* Insert the new keybind:<br />
<keybind key="XF86MenuKB"><br />
<action name="ShowMenu"><br />
<menu>root-menu</menu><br />
</action><br />
</keybind><br />
<br />
===== Intuitive Openbox configuration template =====<br />
Prerequisites:<br />
*mapped 'Pandora key' as 'Windows key'(see above)<br />
*optional: use tint2 as panel<br />
<br />
Features:<br />
*Changing window by '''[Ctrl] + [up]/[down]'''<br />
*Changing workspace by '''[Ctrl] + [left]/[right]'''<br />
*Closing window by '''[Ctrl] + [Q]'''<br />
*Screenshot (window): '''[Ctrl] + [Shift] + [Alt] + [F11]'''<br />
*Screenshot (whole desktop): '''[Ctrl] + [Shift] + [Alt] + [F12]'''<br />
*and some more<br />
<br />
'''Download:'''<br />
[https://www.dropbox.com/s/ud0muusrxq9g9va/rc.xml rc.xml]<br />
<br />
===== Intuitive Openbox menu template =====<br />
Features:<br />
*Most important programms in root menu<br />
*menu entries for WLAN, USB host and low power mode<br />
'''Download:'''<br />
[https://www.dropbox.com/s/fqjxg5xoymg5fep/menu.xml menu.xml]<br />
<br />
== Mounting internal NAND to /mnt ==<br />
<br />
Pandora internal NAND exposes a raw flash interfaces. For such devices, the<br />
Linux kernel exposes a special MTD abstraction and "mtd aware" filesystems may<br />
give a better performance. Pandora uses UBIFS, which needs UBI volume<br />
management to work.<br />
<br />
# Install mtd-utils<br />
# Run the following commands<br />
<br />
<syntaxhighlight lang="bash"><br />
# Attach mtd device 4 to UBI volume management 0<br />
ubiattach -m 4 -d 0<br />
# Mount first partition of the UBI group 0 to /mnt<br />
mount -t ubifs /dev/ubi0_0 /mnt<br />
# Attach mtd device 3 to UBI volume management 1<br />
ubiattach -m 3 -d 1<br />
# Mount first partition of the UBI group 1 to /mnt<br />
mount -t ubifs /dev/ubi1_0 /mnt/boot<br />
</syntaxhighlight><br />
<br />
== Old releases ==<br />
<br />
=== Mark2 minimal-hf2014-01-02 ===<br />
<br />
{| class="wikitable"<br />
|-<br />
! Version !! Description !! Included !! Link !! Checksums (sha256)<br />
|-<br />
| minimal || pandian without any Desktop environment but with the basic pandian package || nano || [http://www.filedropper.com/pandian-mark2-minimal-hf2014-01-02 filedropper: pandian-mark2-minimal-hf2014-01-02]<br />[http://pandian.openpandora.org/releases/pandian-mark2-minimal-hf_2014-01-02.7z pandian.openpandora.org: pandian-mark2-minimal-hf_2014-01-02.7z] || pandian-mark2-minimal-hf_2014-01-02.'''7z'''↓ a35106473e54c55d64c7eae2088d8c267b605baf17d5338b8e2938dbfe3c01ca<br />pandian-mark2-minimal-hf_2014-01-02.'''img'''↓ c68a42c1aef0238c1dd6ff33b628a2c0dea812a7b5b2d906afb185f93f940f84<br />
|-<br />
| minimal-networking fix|| HotFix for pandian-mark2-minimal-hf_2014-01-02 [[#Network_Fix_for_pandian-minimal|Instructions]] || dhcp, network manager || [http://www.filedropper.com/pandian-mark2-minimal-networking2014-01-02tar filedropper: pandian-mark2-minimal-networking_2014-01-02.tar.gz]<br />[http://pandian.openpandora.org/releases/pandian-mark2-minimal-networking_2014-01-02.tar.gz pandian.openpandora.org: pandian-mark2-minimal-networking_2014-01-02.tar.gz] ||pandian-mark2-minimal-networking_2014-01-02.'''tar.gz'''↓ 4cf802391360e5bf183e9d4b5fe8b6f407b3e476baec7b5b0aa7340796680ae6<br />
|}<br />
<br />
===== Network Fix for pandian-minimal =====<br />
If you download the '''pandian-mark2-minimal-hf_2014-01-02''' image there is a lack of tools for setup wifi networking. <br /><br />
'''Note: For the actual download of pandian-minimal, wifi-network-tools are included'''<br />
* Therefore download the fix and place it on the SD-Card where pandian is installed. (you need to have local superuser privileges on your own system as the SD contents aren't owned by your user)<br />
* Boot pandian-minimal<br />
* login with user: "root"<br />
* switch to root-directory: "cd /"<br />
* untar the network-fix: "tar -xf /path-where-file-is-located/pandian-mark2-minimal-networking_2014-01-03.tar.gz"<br />
* setup the fix with command "pdMinimalNetwork"<br />
<br />
* after the script is finished, run "ceni" and setup your wifi-connection<br />
<br />
=== 01 ===<br />
'''Full debian GNU/Linux 7.2 codename Wheezy (stable) with LXDE desktop environment <br /><br />
hardfloat armv7l released Date: 29.09.2013'''<br />
[http://boards.openpandora.org/topic/14482-pandian-01/ forum link]<br />
===Features===<br />
<br />
* Hard-Float-ABI<br />
* Ext4 Root for better performance<br />
* Working Touch-Screen<br />
* Working Nubs<br />
* Screen-Off on LID-Close<br />
* Low-Power-Mode<br />
* WLAN<br />
* Music ( with Sonata/MPD )<br />
* Automount<br />
<br />
=== Upcoming/Ideas ===<br />
<br />
* Assistant for tzdata<br />
* libpnd - PND-Support<br />
* OpenGL not tested yet<br />
* Aptitude<br />
* Minimal-Network-Installation<br />
* Pandian-Debian-Repository for special packages ( libpnd, touch-screen-calibrator )<br />
* of course, in-system-documentation ( Documentation on the pandora )<br />
<br />
Download [http://www.share-online.biz/dl/TPMFFXTM2AU here]<br />
<br />
[http://boards.openpandora.org/topic/14482-pandian-01 Forum thread]<br />
<br />
=== Installation ===<br />
<br />
note: It is assumed here, that the card-reader load the card as /dev/sdc.<br />
Please check before which device is your SD-card !<br />
This can be checked with gparted. Minimum requirement 2GB SD-card.<br />
<br />
# Download Image ( as 7z )<br />
# Verify md5-checksum: <br /> pandian-stable-hf_2013-09-29.001.7z '''2a85bbb40bbe7c6e8f92f058e150951a''' <br /> pandian-stable-hf_2013-09-29.001.img '''e62d101dc708de49af1ca7743e2df2bd'''<br />
# Unzip with 7z ( in Linux with 7zr e <image.7z> )<br />
# Copy image to SD-Card ( dd if=<image.img> of=/dev/sdx ) Don't specify "bs="<br />
<br />
Username: ''Pandian'' <br /><br />
Root password: ''Root'' (change after intsallation as per instructions below)<br />
<br />
#Open a terminal<br />
#Type "su"<br />
#Type "passwd"<br />
#Enter your new root password when prompted, and then again to confirm.<br />
#Dont forget your password<br />
<br />
=== Resize ===<br />
Default installation is limited to 2GB, if you want to expand it, you can, provided you have installed it on a bigger SD-card and want to make use of the space.<br />
<br />
* Size will be changed with parted<br />
* parted /dev/sdx<br />
* show size with "print free"<br />
<br />
# (parted) print free # Model: USB Mass Storage Device (scsi)<br />
# Disk /dev/sdc: 7965MB# Sector size (logical/physical): 512B/512B# Partition Table: msdos<br />
<br />
{| class="wikitable"<br />
|-<br />
| Number || Start || End || Size || Type || File system || Flags<br />
|-<br />
| || 32,3kB || 1049kB || 1016kB || || Free Space ||<br />
|-<br />
| 1 || 1049kB || 53,5MB || 52,4MB || primary || fat32 ||<br />
|-<br />
| 2 || 53,5MB || 1933MB || 1879MB || primary || ext4 ||<br />
|-<br />
| || 1933MB || 7965MB || 6032MB || || Free Space ||<br />
|}<br />
<br />
we would like to resize the second partition to the nearly end ( -512MB for swap )<br />
# (parted) resizepart # Partition number? 2 # Warning: Partition /dev/sdc2 is being used. Are you sure you want to continue?# Yes/No? YES # End? [1933MB]? 7453MB this is how the partitions looks like after<br />
<br />
(parted) print free * Model: USB Mass Storage Device (scsi)* Disk /dev/sdc: 7965MB* Sector size (logical/physical): 512B/512B* Partition Table: msdos<br />
<br />
{| class="wikitable"<br />
|-<br />
| Number || Start || End || Size || Type || File system || Flags<br />
|-<br />
| || 32,3kB || 1049kB || 1016kB || || Free Space ||<br />
|-<br />
| 1 || 1049kB || 53,5MB || 52,4MB || primary || fat32 ||<br />
|-<br />
| 2 || 53,5MB || '''7453'''MB || '''7400'''MB || primary || ext4 ||<br />
|-<br />
| || '''7453'''MB || 7965MB || '''512'''MB || || Free Space ||<br />
|}<br />
<br />
create Swap (optional)<br />
<br />
(parted) mkpart primary linux-swap<br />
Start? 7453MB <br />
End? 7965MB<br />
<br />
this is how the partitions looks like after<br />
<br />
(parted) print free * Model: USB Mass Storage Device (scsi)* Disk /dev/sdc: 7965MB* Sector size (logical/physical): 512B/512B* Partition Table: msdos<br />
<br />
<br />
{| class="wikitable"<br />
|-<br />
| Number || Start || End || Size || Type || File system || Flags<br />
|-<br />
| || 32,3kB || 1049kB || 1016kB || || Free Space ||<br />
|-<br />
| 1 || 1049kB || 53,5MB || 52,4MB || primary || fat32 ||<br />
|-<br />
| 2 || 53,5MB || 7453MB || 7400MB || primary || ext4 ||<br />
|-<br />
| || '''7453MB''' || '''7453MB''' || '''278kB''' || || '''Free Space'''||<br />
|-<br />
| || 7453MB || 7965MB || 512MB || '''primary''' || Free Space ||<br />
|}<br />
disregard the 4th entry, the new resized partition is the last one.<br />
<br />
resize filesystem<br />
*resize2fs /dev/sdc2<br />
<br />
create swap<br />
<br />
*mkswap /dev/sdc3<br />
<br />
[[Category:Documentation]]<br />
[[Category:Operating system]]<br />
[[Category:Software]]</div>Vinipsmakerhttps://pandorawiki.org/index.php?title=Pandian&diff=29518Pandian2014-02-13T06:59:15Z<p>Vinipsmaker: /* Pandian - MARK 2 The newest release */</p>
<hr />
<div>[[File:Post-3032-0-90501200-1380479048.png|thumbnail|Logo]]<br />
<br />
'''Pandian is an debian-based armhf desktop and CLI distro for your Pandora.'''<br />
Ideas, Bugs, or feature-requests? Report to:<br />
http://www.evilbrain.de/projects/pandian/issues<br />
<br />
=== Main Features ===<br />
* The debian way.<br />
* Access to all packages from Debian software repositories.<br />
* Pandian hosts its own Debian repo http://pandian.openpandora.org with up to date scripts, all settings and extra programs.<br />
* armhf-build, which means '''f'''loat operations are relegated to special '''h'''ardware rather than taxing the main cpu with them (as with softfloat)<br />
<br />
== '''Pandian''' - '''MARK 2''' The newest release ==<br />
Second version of Pandian<br /><br />
'''Full Debian GNU/Linux based on sid branch with LXDE desktop environment (normal) or CLI (minimal-) build.''' <br /> <br />
'''hardfloat armv7l''' yule release '''Date: 29.12.2013'''<br />
[http://boards.openpandora.org/topic/15302-pandian-mark2/ forum link]<br />
<br />
=== '''Installation''' requirements===<br />
* '''Pandora''' ;) All versions fine?<br />
* '''SD card''' minimum 2GB ( was tested with 4GB )<br />
* '''PC''' Linux|GNU, BSD, Mac, or Windows for putting the install on the aforementioned SD-card.<br />
<br />
==== '''Download''' a version of Pandian ====<br />
There are two different versions of the "MARK2" release:<br />
{| class="wikitable"<br />
|-<br />
! Version !! Description !! Includes !! Link !! Size !! Checksums (sha256)<br />
|-<br />
| '''base''' || pandian for advanced users, without any Desktop environment but with basic network-tools ( iw, ceni ) || ceni ( network-manager) network-manager MediaPlayer ( mpd ) iw (wireless-tools) || [http://www.filedropper.com/pandian-mark2-base-hf2014-01-05 filedropper: pandian-mark2-base-hf2014-01-05.7z]<br />[http://pandian.openpandora.org/releases/pandian-mark2-base-hf_2014-01-05.7z pandian.openpandora.org: pandian-mark2-base-hf_2014-01-05.7z] || 161MB || pandian-mark2-base-hf_2014-01-05.'''7z'''↓ b11ffe9cc4fd6111775d0b48c133b1e1afcf6865de784b9865db0cea29674d85<br />pandian-mark2-base-hf_2014-01-05.'''img'''↓ c1903423a5b5e0d04563f14526e8161fbe77a102e64d5d571bcf3b87870fa211 <br />
|-<br />
| '''normal''' || pandian for regular users, with full Desktop environment (LXDE) || network-manager MediaPlayer ( mpd with gurlp ) nub-support calibrator-application medit Text editor. mpd audio player backend, glurp as frontend || [http://www.filedropper.com/pandian-mark2-hf2014-01-02 filedropper: pandian-mark2-hf2014-01-02]<br />[http://pandian.openpandora.org/releases/pandian-mark2-hf_2014-01-02.7z pandian.openpandora.org: pandian-mark2-hf_2014-01-02.7z] || 394MB || pandian-mark2-hf_2014-01-02.'''7z'''↓ 3731d70a4d8ec59e79a046f34f69c5c809be73e38cfc91a5b580478b8f8c222f<br />pandian-mark2-hf_2014-01-02.'''img'''↓ 83b8d3fa5bd0588cfb8b11cfe6b827819e6311e98396a6e32c46c9692f072e96<br />
|}<br />
Notes:<br />
* It is trivial to change from LXDE to [[Xfce]] (or any other desktop environment), adding a new user interface atop the minimal install is more advanced as manual setup of USB-Networking or wifi is needed to do so.<br />
* All images have automount ( for USB-Sticks, external Hard-Drives etc. ) and ALSA sound support<br />
<br />
==== '''Verify''' that your download is OK ====<br />
* After you download the version you picked, verify its checksum to make sure the integrity of the downloaded file matches (in being an exact copy the one on the server)<br />
<br />
#open up a terminal<br />
# "'''sha256sum ~/downloads/pandian/<image.7Z>'''" (input where you have the file and the one you chose to download)<br />
# make sure it matches the given md5sum, if not, try to download another copy. This step isnt mandatory, but its there to make sure everything is fine before we proceed.<br />
<br />
==== '''Unzip''' the downloaded image ====<br />
*Unzipping 7z archives<br />
# Open up a terminal (or do it your way)<br />
# Enter"'''7zr e <image.7z>'''" <br />
# You now have an unzipped image<br />
<br />
==== (Optional) '''Preparation''', find out where to install ====<br />
<br />
This step helps you find out where the SD card is mounted.<br /><br />
<br />
If you have an build in SD-Card-Reader this is normally /dev/mmcblk0p1 ( in Debian )<br />
If you have an USB-Card-Reader this is something like /dev/sdc ( in Debian )<br />
* Open an Terminal and input "'''lsblk'''"<br />
* You will get a list with the actual Block-Devices<br />
* The SD-Card should be of type 'disk'<br />
<br />
You can also try gparted to see which devices you have and identify the SD-card from there<br />
<br />
==== '''Install''' the image to the SD-Card ====<br />
{{warning<br />
| Warning, this will destroy everything on the device ! So please be very sure which device is you SD-Card !<br />
}}<br />
# Here it will be /dev/sd'''x''' <br />
<br />
(note that the x after the sd is the drive letter, it could be ''sd'''b''''' sd'''c''''' or ''sd'''d''''' and so on, lets say its sd'''c'''. Which means c is the drive letter for the SD-card and that sd- a and -b are in use for other things. <br /><br />
<br /><br />
When writing to the device, make sure not to include the following number, which is the partition ''sdc'''1''''' ''sdc'''2''''' etc. since we are using the base card here) <br /><br />
<br /><br />
This means considerations for partitions and filesystem are taken care of, so no need to worry about that. Do ''not'' pick the wrong drive however, since dd will happily write the image to somewhere it isn't supposed to if you just give it root privileges and hit enter.)<br /><br />
<br /><br />
#Copy image to SD-Card with "'''dd if=<image.img> of=/dev/sdx'''"<br />
<br />
You now have Pandian installed on your SD card and its ready to go. :) However after copying the image to the SD-Card resizing the partitionspartitions is needed to fully utilize the space, since by default the install is limited to 2GB. (If your card is bigger Pandian is still limited to those 2GB)<br />
<br />
====(Optional) '''Resize''' SD-Card ====<br />
<br />
* This can be done with gparted on linux|gnu<br />
* start gparted<br />
<br />
[[File:Gparted resize 001.png|800px|thumbnail|none|The actual partitions of the SD-CARD. Please select the correct device at the top right corner.]]<br />
[[File:Gparted resize 002.png|800px|thumbnail|none|Rightclick to the second partition and select Resize/Move]]<br />
[[File:Gparted resize 003.png|400px|thumbnail|none|You see the actual Size of this partition]]<br />
[[File:Gparted resize 004.png|400px|thumbnail|none|Move the slider to the right, or set "Free space following" to 0]]<br />
[[File:Gparted resize 005.png|800px|thumbnail|none|After you press the "Resize/Move" button, you get the preview of the new partition structure]]<br />
[[File:Gparted resize 006.png|800px|thumbnail|none|When you press "Apply all operations", youve got a warning that you can loss your Data.<br />THIS IS THE LAST CHANGE TO CHECK IF YOU SELECTED THE CORRECT DEVICE. <br />
<br />
<br />Press Apply if everything is correct.]]<br />
<br />
* After everything is fine, you have a fresh install of Pandian on you SD-Card<br />
<br /><br />
<br />
=== '''Startup''' wizard ===<br />
* Upon first boot of Pandian, there is a first-start-wizard which guides you through some basic settings.<br />
These are: Locale, timezone, time and date, password for admin and user setup and finally calibration of the touchscreen.<br />
The script is located in /usr/sbin/pdSetup should you need to run it again.<br />
<br />
==== Navigation ====<br />
* Inside a dialog you can navigate with the cross/dpad<br />
* Hit space to select a checkbox<br />
* Moving between <Ok> and <Cancel> can be down with the use of TAB ( Tap Fn then tap Space )<br />
* Acknowledge with Enter<br />
<br />
*All done, enjoy :)<br />
<br />
[[File:PandianDesktop.png|frameless]]<br />
<br />
=== '''Network''' connection instructions:===<br />
<br />
====normal ====<br />
* Use network-manager<br />
<br />
====minimal====<br />
* Use ceni network manager<br />
<br />
===(Optional) '''Update''' of Pandian system ===<br />
* After you setup your Pandora, it is useful to update your system. It is a rolling release, which means since the image was made, the system is forever updated with new packages.<br />
* open a terminal<br />
* get root ( command is: "su" )<br />
* enter "apt-get update" to download package information from the repositories<br />
* enter "apt-get upgrade" to update all packages that are out of date<br />
(pandian packages are also updated this way :)<br />
* Note: to update only the pandian-packages enter "apt-get upgrade pandian*"<br />
<br />
<br />
===== Upgrade from "minimal" to "normal" =====<br />
It is possible to upgrade from the "minimal"-image to the "normal"-image.<br />
This can simlpy be done by installing the pandian-lxde debian package<br />
* get an connection to the internet(tm) with ceni<br />
* open an terminal<br />
* get root ( with command: "su" )<br />
* update package list with: "apt-get update"<br />
* install pandian-lxde with "apt-get install pandian-lxde"<br />
* after everything is finished login with the user you setup before<br />
* open an terminal<br />
* get root ( with command: "su" )<br />
* load keymap with "xmodmap /etc/skel/.pndXmodmap"<br />
* the easiest way is to activate the first-start-wizard with "insserv firstuse"<br />
* reboot and go through the wizard<br />
<br />
===(Optional) '''Information''' ===<br />
<br />
=== Additional Users/Groups ===<br />
There are additional Users/Groups added to the standard debian-users<br />
==== Groups ====<br />
* admin<br />
:This superuser privilege is used by sudo for admin task like shutdown etc.<br />
* mpd<br />
:This user/group has access to mpd-related files ( socket, pid-files )<br />
<br />
==== Power-Modes ====<br />
===== LID-Close =====<br />
* If the lid is closed only the screen will be disabled<br />
===== Low-Power-Mode / Shutdown =====<br />
[[File:Lead Photo For Power modes0-2920264867079787.jpg|thumbnail|right|Switch]]<br />
* Switch to left: Pandora will go to low power mode ( reduce CPU, Disable LID, Pause processes )<br />
* Switch to right: Shutdown the pandora<br />
<br />
==== pdcmd ====<br />
* This script handles all system-related stuff, like lid, low-power, usb or wifi.<br />
* This should run as root<br />
<br /><br />
Usage:<br />
{| class="wikitable"<br />
|-<br />
! Command !! What's done<br />
|-<br />
| /usr/sbin/pdcmd LCDBright + || Increase the light of the lid<br />
|-<br />
| /usr/sbin/pdcmd LCDBright - || Decrease the light of the lid<br />
|-<br />
| /usr/sbin/pdcmd LCDBright 10 || Set the Brightness of the lid to 10<br />
|-<br />
| /usr/sbin/pdcmd LCDTrigger || Trigger the lid ( WARNING: THIS DISABLES THE LID ! )<br />
|-<br />
| /usr/sbin/pdcmd USBNetTrigger || Enable the USB-Networking (module g_ether). If usbnetworking is on, it will disable it<br />
|-<br />
| /usr/sbin/pdcmd USBHostTrigger || Enable the USB-Host (module ehci-hcd). If USB-Host is on, it will disable it<br />
|-<br />
| /usr/sbin/pdcmd WLANTrigger || Enable Wifi. If Wifi is enabled it will be disabled<br />
|}<br />
<br />
==== Several tweaks ====<br />
===== Making the Pandora Button work like the Windows key =====<br />
Many users complained, their Pandora Button has no function in Pandian. This can be changed by mapping the button as ''"mod4"''. <br />
*Make sure, the file .pndXmodmap is read in on startup. Open the file ''.xsessionrc'' in your ''home'' directory and, if not existent, add following to the last line:<br />
xmodmap /home/yourUsername/.pndXmodmap<br />
*There must be the absolute path (no "~/").<br />
*Insert in the last line of the .pndXmodmap from your ''home'' directory:<br />
add mod4 = XF86MenuKB<br />
<br />
===== Openbox =====<br />
Pandian comes with an unconfigured Openbox as a possible session. Here are some config templates.<br />
<br />
====== Use the Pandora key to start the menu ======<br />
*If not existent,<br />
cp /etc/xdg/openbox/rc.xml /home/yourUserName/.config/openbox/<br />
* Open the rc.xml with a text editor and search for the section <keyboard>.<br />
* Insert the new keybind:<br />
<keybind key="XF86MenuKB"><br />
<action name="ShowMenu"><br />
<menu>root-menu</menu><br />
</action><br />
</keybind><br />
<br />
===== Intuitive Openbox configuration template =====<br />
Prerequisites:<br />
*mapped 'Pandora key' as 'Windows key'(see above)<br />
*optional: use tint2 as panel<br />
<br />
Features:<br />
*Changing window by '''[Ctrl] + [up]/[down]'''<br />
*Changing workspace by '''[Ctrl] + [left]/[right]'''<br />
*Closing window by '''[Ctrl] + [Q]'''<br />
*Screenshot (window): '''[Ctrl] + [Shift] + [Alt] + [F11]'''<br />
*Screenshot (whole desktop): '''[Ctrl] + [Shift] + [Alt] + [F12]'''<br />
*and some more<br />
<br />
'''Download:'''<br />
[https://www.dropbox.com/s/ud0muusrxq9g9va/rc.xml rc.xml]<br />
<br />
===== Intuitive Openbox menu template =====<br />
Features:<br />
*Most important programms in root menu<br />
*menu entries for WLAN, USB host and low power mode<br />
'''Download:'''<br />
[https://www.dropbox.com/s/fqjxg5xoymg5fep/menu.xml menu.xml]<br />
<br />
Pandora internal NAND exposes a raw flash interfaces. For such devices, the<br />
Linux kernel exposes a special MTD abstraction and "mtd aware" filesystems may<br />
give a better performance. Pandora uses UBIFS, which needs UBI volume<br />
management to work.<br />
<br />
# Install mtd-utils<br />
# Run the following commands<br />
<br />
<code><br />
# Attach mtd device 4 to UBI volume management 0<br />
ubiattach -m 4 -d 0<br />
# Mount first partition of the UBI group 0 to /mnt<br />
mount -t ubifs /dev/ubi0_0 /mnt<br />
# Attach mtd device 3 to UBI volume management 1<br />
ubiattach -m 3 -d 1<br />
# Mount first partition of the UBI group 1 to /mnt<br />
mount -t ubifs /dev/ubi1_0 /mnt/boot<br />
</code><br />
<br />
== Old releases ==<br />
<br />
=== Mark2 minimal-hf2014-01-02 ===<br />
<br />
{| class="wikitable"<br />
|-<br />
! Version !! Description !! Included !! Link !! Checksums (sha256)<br />
|-<br />
| minimal || pandian without any Desktop environment but with the basic pandian package || nano || [http://www.filedropper.com/pandian-mark2-minimal-hf2014-01-02 filedropper: pandian-mark2-minimal-hf2014-01-02]<br />[http://pandian.openpandora.org/releases/pandian-mark2-minimal-hf_2014-01-02.7z pandian.openpandora.org: pandian-mark2-minimal-hf_2014-01-02.7z] || pandian-mark2-minimal-hf_2014-01-02.'''7z'''↓ a35106473e54c55d64c7eae2088d8c267b605baf17d5338b8e2938dbfe3c01ca<br />pandian-mark2-minimal-hf_2014-01-02.'''img'''↓ c68a42c1aef0238c1dd6ff33b628a2c0dea812a7b5b2d906afb185f93f940f84<br />
|-<br />
| minimal-networking fix|| HotFix for pandian-mark2-minimal-hf_2014-01-02 [[#Network_Fix_for_pandian-minimal|Instructions]] || dhcp, network manager || [http://www.filedropper.com/pandian-mark2-minimal-networking2014-01-02tar filedropper: pandian-mark2-minimal-networking_2014-01-02.tar.gz]<br />[http://pandian.openpandora.org/releases/pandian-mark2-minimal-networking_2014-01-02.tar.gz pandian.openpandora.org: pandian-mark2-minimal-networking_2014-01-02.tar.gz] ||pandian-mark2-minimal-networking_2014-01-02.'''tar.gz'''↓ 4cf802391360e5bf183e9d4b5fe8b6f407b3e476baec7b5b0aa7340796680ae6<br />
|}<br />
<br />
===== Network Fix for pandian-minimal =====<br />
If you download the '''pandian-mark2-minimal-hf_2014-01-02''' image there is a lack of tools for setup wifi networking. <br /><br />
'''Note: For the actual download of pandian-minimal, wifi-network-tools are included'''<br />
* Therefore download the fix and place it on the SD-Card where pandian is installed. (you need to have local superuser privileges on your own system as the SD contents aren't owned by your user)<br />
* Boot pandian-minimal<br />
* login with user: "root"<br />
* switch to root-directory: "cd /"<br />
* untar the network-fix: "tar -xf /path-where-file-is-located/pandian-mark2-minimal-networking_2014-01-03.tar.gz"<br />
* setup the fix with command "pdMinimalNetwork"<br />
<br />
* after the script is finished, run "ceni" and setup your wifi-connection<br />
<br />
=== 01 ===<br />
'''Full debian GNU/Linux 7.2 codename Wheezy (stable) with LXDE desktop environment <br /><br />
hardfloat armv7l released Date: 29.09.2013'''<br />
[http://boards.openpandora.org/topic/14482-pandian-01/ forum link]<br />
===Features===<br />
<br />
* Hard-Float-ABI<br />
* Ext4 Root for better performance<br />
* Working Touch-Screen<br />
* Working Nubs<br />
* Screen-Off on LID-Close<br />
* Low-Power-Mode<br />
* WLAN<br />
* Music ( with Sonata/MPD )<br />
* Automount<br />
<br />
=== Upcoming/Ideas ===<br />
<br />
* Assistant for tzdata<br />
* libpnd - PND-Support<br />
* OpenGL not tested yet<br />
* Aptitude<br />
* Minimal-Network-Installation<br />
* Pandian-Debian-Repository for special packages ( libpnd, touch-screen-calibrator )<br />
* of course, in-system-documentation ( Documentation on the pandora )<br />
<br />
Download [http://www.share-online.biz/dl/TPMFFXTM2AU here]<br />
<br />
[http://boards.openpandora.org/topic/14482-pandian-01 Forum thread]<br />
<br />
=== Installation ===<br />
<br />
note: It is assumed here, that the card-reader load the card as /dev/sdc.<br />
Please check before which device is your SD-card !<br />
This can be checked with gparted. Minimum requirement 2GB SD-card.<br />
<br />
# Download Image ( as 7z )<br />
# Verify md5-checksum: <br /> pandian-stable-hf_2013-09-29.001.7z '''2a85bbb40bbe7c6e8f92f058e150951a''' <br /> pandian-stable-hf_2013-09-29.001.img '''e62d101dc708de49af1ca7743e2df2bd'''<br />
# Unzip with 7z ( in Linux with 7zr e <image.7z> )<br />
# Copy image to SD-Card ( dd if=<image.img> of=/dev/sdx ) Don't specify "bs="<br />
<br />
Username: ''Pandian'' <br /><br />
Root password: ''Root'' (change after intsallation as per instructions below)<br />
<br />
#Open a terminal<br />
#Type "su"<br />
#Type "passwd"<br />
#Enter your new root password when prompted, and then again to confirm.<br />
#Dont forget your password<br />
<br />
=== Resize ===<br />
Default installation is limited to 2GB, if you want to expand it, you can, provided you have installed it on a bigger SD-card and want to make use of the space.<br />
<br />
* Size will be changed with parted<br />
* parted /dev/sdx<br />
* show size with "print free"<br />
<br />
# (parted) print free # Model: USB Mass Storage Device (scsi)<br />
# Disk /dev/sdc: 7965MB# Sector size (logical/physical): 512B/512B# Partition Table: msdos<br />
<br />
{| class="wikitable"<br />
|-<br />
| Number || Start || End || Size || Type || File system || Flags<br />
|-<br />
| || 32,3kB || 1049kB || 1016kB || || Free Space ||<br />
|-<br />
| 1 || 1049kB || 53,5MB || 52,4MB || primary || fat32 ||<br />
|-<br />
| 2 || 53,5MB || 1933MB || 1879MB || primary || ext4 ||<br />
|-<br />
| || 1933MB || 7965MB || 6032MB || || Free Space ||<br />
|}<br />
<br />
we would like to resize the second partition to the nearly end ( -512MB for swap )<br />
# (parted) resizepart # Partition number? 2 # Warning: Partition /dev/sdc2 is being used. Are you sure you want to continue?# Yes/No? YES # End? [1933MB]? 7453MB this is how the partitions looks like after<br />
<br />
(parted) print free * Model: USB Mass Storage Device (scsi)* Disk /dev/sdc: 7965MB* Sector size (logical/physical): 512B/512B* Partition Table: msdos<br />
<br />
{| class="wikitable"<br />
|-<br />
| Number || Start || End || Size || Type || File system || Flags<br />
|-<br />
| || 32,3kB || 1049kB || 1016kB || || Free Space ||<br />
|-<br />
| 1 || 1049kB || 53,5MB || 52,4MB || primary || fat32 ||<br />
|-<br />
| 2 || 53,5MB || '''7453'''MB || '''7400'''MB || primary || ext4 ||<br />
|-<br />
| || '''7453'''MB || 7965MB || '''512'''MB || || Free Space ||<br />
|}<br />
<br />
create Swap (optional)<br />
<br />
(parted) mkpart primary linux-swap<br />
Start? 7453MB <br />
End? 7965MB<br />
<br />
this is how the partitions looks like after<br />
<br />
(parted) print free * Model: USB Mass Storage Device (scsi)* Disk /dev/sdc: 7965MB* Sector size (logical/physical): 512B/512B* Partition Table: msdos<br />
<br />
<br />
{| class="wikitable"<br />
|-<br />
| Number || Start || End || Size || Type || File system || Flags<br />
|-<br />
| || 32,3kB || 1049kB || 1016kB || || Free Space ||<br />
|-<br />
| 1 || 1049kB || 53,5MB || 52,4MB || primary || fat32 ||<br />
|-<br />
| 2 || 53,5MB || 7453MB || 7400MB || primary || ext4 ||<br />
|-<br />
| || '''7453MB''' || '''7453MB''' || '''278kB''' || || '''Free Space'''||<br />
|-<br />
| || 7453MB || 7965MB || 512MB || '''primary''' || Free Space ||<br />
|}<br />
disregard the 4th entry, the new resized partition is the last one.<br />
<br />
resize filesystem<br />
*resize2fs /dev/sdc2<br />
<br />
create swap<br />
<br />
*mkswap /dev/sdc3<br />
<br />
[[Category:Documentation]]<br />
[[Category:Operating system]]<br />
[[Category:Software]]</div>Vinipsmakerhttps://pandorawiki.org/index.php?title=New_PND_format&diff=29504New PND format2014-02-11T03:19:21Z<p>Vinipsmaker: another option to home-dir</p>
<hr />
<div>The current PND format has some shortcomings as listed below. This page should serve as a discussion page/white board for how the format could be improved.<br />
<br />
''' Proposal for discussion, this is just some opening shots '''<br />
<br />
== Current situation ==<br />
The current (ISO-based) PND format has the following shortcomings:<br />
<br />
* It currently uses ISO, SquashFS, and other filesystems... This means that there's no standard to follow, and libpnd needs to carry around support for a multitude of file systems.<br />
* It (often, but not always) uses the [http://en.wikipedia.org/wiki/ISO_9660 ISO file system] which is inefficient at storing data, because:<br />
** It contains duplicate headers for each file entry[http://users.telenet.be/it3.consultants.bvba/handouts/ISO9960.html] (one version with big-endian integers, one version with little-endian integers[http://en.wikipedia.org/wiki/Endianness]) which leads to many kilobytes of wasted space.<br />
** Its file tables are fixed-size, and there are therefore limitations on how many folders you can have, what names you can give your files, how big your files can be etc. For instance, files can only have names that are max 31 characters long, all upper-case, and limited to the ASCII character encoding, and an ISO does only support a folder depth of 8[http://www.on-time.com/rtos-32-docs/rtfiles-32/programming-manual/iso-9660/]. To support more, the Joliet file system extension is needed (or isofs won't recognize all file paths), and the Joliet header can be placed practically anywhere in the file, which means additional seek times.<br />
** It needs various file system extensions to behave correctly, like for example [http://en.wikipedia.org/wiki/Rock_Ridge Rock Ridge Interchange Protocol] and [http://en.wikipedia.org/wiki/Joliet_(file_system) Joliet]. Without them, it becomes pretty useless as demonstrated above, and with them, it becomes difficult to read by a tool (You can only use isofs! There are no other tools out there for programmers to use!).<br />
** It's very difficult to use since you need special ISO making tools to create the image. ISO is a relatively rare format if you're not technically inclined.<br />
** If you want to add or remove files from it, it's near impossible if you use a compact ISO file, since there's no way that you can "expand" an existing ISO file.<br />
* The PND header data is at the end (or in the middle of the file if a screenshot is included) which makes it impossible for e.g. libmagic to recognize the PND file. It will instead recognize it as an ISO file. Having the header data at the end also makes it take a very long time to find the data, making tools and the libpnd library very inefficient.<br />
* The PND file uses a custom XML format for its metadata. There's no reason to do this, especially since the established ".desktop" file format fills exactly the same function.<br />
(note: ".desktop" does not contain all metadata we have wanted over time, but it's of course possible to add extensions à la "X-Pandora-Whatever=xyz")<br />
* There's no "index" for the PND file. The whole file has to be scanned (albeit backwards) to find a PXML file, and there's a big chance for false positives etc.<br />
* Data is just appended linearly to the file so there's no order. If the format is to be extended (to e.g. include an icon file after the screenshot file), should the data just be appended as well?<br />
* UTF-8 is strictly the only encoding that is supported. If you make your PXML on a Windows machine, it won't work.<br />
<br />
== Proposed revisions ==<br />
=== Step 1: File system ===<br />
The file system for PNDs should be replaced by the ''uncompressed'' ZIP archive format. ZIP has the advantage that it's incredibly compact, and uncompressed ZIP makes it possible to read data from the file without having to do any decompression.<br />
<br />
ZIP files are mountable using various implementations of zipfs, and most of these implementations won't store files in memory when they are being read if the ZIP file is uncompressed.<br />
<br />
=== Step 2: Metadata ===<br />
PND files should no longer require special tools for them to be created. Therefore, since ZIP files support random access on files, it shouldn't be necessary to append/prepend metadata to the file. The user simply includes the "PXML.xml" and "preview.png" files inside of the ZIP, and any tools that need information about the PND can simply go to the central directory of the ZIP file (or use a simple ZIP library to do it for them) and get the location of the file inside of the ZIP. This should also dramatically decrease loading times for PNDs.<br />
<br />
Note; performance testing is needed; at the time we started, zip was enormously slower than plain ISO; with driver changes, it may or may not be so, hence our adopted multiple-filesystem-type system. zipfs is one possible option.<br />
<br />
=== Step 3: Metadata format ===<br />
Current PND files include so-called "PXML.xml" files. These files have a custom XML format that has a strange structure.<br />
<br />
These files should be replaced by ".desktop" files. A PND can contain one or more ".desktop" files in its root directory that specify how an application should be launched. PND tools simply use all ".desktop" files they can find in the PND when creating launchers for the contents of the PND.<br />
<br />
== Advantages ==<br />
* There don't have to be ''any'' special tools for reading PND files. The package can be run on any platform using any programming language that can read ZIP files.<br />
* We can use existing facilities to manage the launching of applications. The ".desktop" files can basically be copied without modification into standard locations of the system, and all launchers will become aware of them.<br />
* Reading PND files becomes easier (because of better tool support... sorry, but you can't link to libpnd in all programming languages) and quicker (becuase of the central directory for random file access).<br />
<br />
=== Benchmarks ===<br />
A benchmark was carried through to measure the performance of various package implementations.<br />
The compared systems were:<br />
* ZIP-based uncompressed packages using fuse-zip (a zipfs implementation) to read the files<br />
* ZIP-based compressed packages also using fuse-zip.<br />
* ISO8859-based packages using isofs<br />
* CramFS-based packages.<br />
<br />
A total of 9 files were generated for the test. They have sizes 1, 2, 3, 4, 16, 32, 64, 128 and 256 MB, and were generated via "dd if=/dev/urandom of=$file bs=1M count=$size".<br />
<br />
Because the files contain random data, they did not compress very well. The resulting files were:<br />
* "zipcompressed.zip" 511.1 MB<br />
* "zipuncompressed.zip" 511.0 MB<br />
* "iso.iso" 511.3 MB<br />
* "cramfs.image" 95.4 MB (WOW!)<br />
<br />
Some notes:<br />
* No write tests were done for obvious reasons<br />
* No random access tests were done since the Pandora will use SD cards and thus not get penalized from random access, and that's not what we're interested in anyways.<br />
* Before each test, the commands "sync; echo 3 > /proc/sys/vm/drop_caches" were run.<br />
<br />
==== First test: linear read time ====<br />
Command: "time cat mountpoint/* > /dev/null"<br />
<br />
* ISO: 8.119 seconds<br />
* CramFS: 1.488 seconds (WOW!)<br />
* Zip (compressed): 8.535 seconds<br />
* Zip (uncompressed): 8.290 seconds<br />
<br />
==== Second test: RAM usage ====<br />
-- TODO, have to find a good method of measuring this --<br />
<br />
== Remaining issues ==<br />
* How should libpnd tell the difference between old and new PND files? Should it depend on the "file" tool, should it run its own recognition algorithm, or what? Should the extension be changed to e.g. ".box" (Which was proposed in a thread)?<br />
* How to make sure that the ZIP files are uncompressed? Should we provide a script that "uncompresses" an ordinary ZIP file?<br />
* Is cramfs better than ZIP?<br />
** cramfs cannot be uncompressed like ZIP can<br />
** If you compare compressed ZIP and cramfs, cramfs is more efficient.<br />
** cramfs is harder to use than ZIP (for the developer).<br />
** cramfs ''needs'' to have all of its opened files in memory.<br />
<br />
== Upgrade path from the old PND format ==<br />
A simple script can be written that extracts the old PND file, including the screenshot and the PXML.xml file. The PXML.xml file is then converted to one or many .desktop files, and the desktop files, the preview.png file, and the package contents are compacted into a ZIP that then is renamed to "*.pnd"<br />
<br />
== Usage scenario ==<br />
=== Repackaging of an application from another package format ===<br />
* The user grabs the package for the application<br />
* He dumps the executable and all required libraries into a folder<br />
* He dumps a "screenshot.png" file into the folder that he's made<br />
* He copies the ".desktop" file(s) for the application from the old package, opens them, replaces "Exec=/usr/bin/bla" with "Exec=./bla" and saves them in the directory<br />
* He uses a ZIP archiver to make a ZIP out of the folder, making sure that he sets the "uncompressed" option.<br />
<br />
=== Creating PNDs as part of a build process ===<br />
* The build tool creates a directory with all of the necessary information like above and invokes the "zip" utility to compress the folder.<br />
<br />
=== Accessing a PND's contents for the... ===<br />
<br />
==== ...lazy programmer ====<br />
* The programmer extracts the PND into a folder via libzip and accesses its contents.<br />
<br />
==== ...pragmatic programmer ====<br />
* The programmer uses zipfs to mount the ZIP and accesses the file's contents.<br />
<br />
==== ...smart performance-aware programmer ====<br />
* The programmer fseek()'s the ZIP file until he finds 0x04034b50.<br />
* He jumps 18 bytes forward and reads the int at that location, storing it in "length".<br />
* He jumps 4 bytes and checks if this int matches "length". If it doesn't, it means that the file is compressed, and an error is reported.<br />
* He then jumps forward 4 bytes and stores the short at that location in a variable "nameLength".<br />
* Another 2-byte jump gets the short "extensionsLength".<br />
* He then jumps 2 bytes and reads "nameLength" amount of bytes from the file.<br />
* He then uses strcmp() to see if this string matches the sought-after file. If not, he continues the fseek().<br />
* The programmer now skips "entensionsLength" amount of bytes.<br />
* The programmer reads "length" amount of bytes from the file, and uses this as the sought-after file data.<br />
<br />
==== ...performance fascist/programmer who wants low seek times ====<br />
* The programmer fseek()'s the file from the end until he finds 0x02014b50.<br />
* He then uses the following table to get the information he needs:<br />
{|class="wikitable"<br />
|+ ZIP central directory file header<br />
|-<br />
! Offset !! Bytes !! Description<br />
|-<br />
| 0 || 4 || Central directory file header signature = 0x02014b50<br />
|-<br />
| 4 || 2 || Version made by<br />
|-<br />
| 6 || 2 || Version needed to extract (minimum)<br />
|-<br />
| 8 || 2 || General purpose bit flag<br />
|-<br />
| 10 || 2 || Compression method<br />
|-<br />
| 12 || 2 || File last modification time<br />
|-<br />
| 14 || 2 || File last modification date<br />
|-<br />
| 16 || 4 || CRC-32<br />
|-<br />
| 20 || 4 || Compressed size<br />
|-<br />
| 24 || 4 || Uncompressed size<br />
|-<br />
| 28 || 2 || File name length (''n'')<br />
|-<br />
| 30 || 2 || Extra field length (''m'')<br />
|-<br />
| 32 || 2 || File comment length (''k'')<br />
|-<br />
| 34 || 2 || Disk number where file starts<br />
|-<br />
| 36 || 2 || Internal file attributes<br />
|-<br />
| 38 || 4 || External file attributes<br />
|-<br />
| 42 || 4 || Relative offset of local file header<br />
|-<br />
| 46 || ''n'' || File name<br />
|-<br />
| 46+''n'' || ''m'' || Extra field<br />
|-<br />
| 46+''n''+''m'' || ''k'' || File comment<br />
|}<br />
<br />
* The relative file offset can then be used to jump to the file in question. The file is then scanned as described in the previous section.<br />
<br />
== The 2014/Pyra effort ==<br />
<br />
A new format is being developed for the DragonBox Pyra handheld. Here are the requirements/notes:<br />
<br />
{| class="wikitable"<br />
|-<br />
! Requirement !! Solutions !! Importance !! Status (decided, undecided, post-proned, won't fix, ...) !! Notes<br />
|-<br />
| Should the new format be supported by Pandora AND Pyra? ||<br />
* Yes<br />
** Makes releasing compatible applications easier.<br />
* no<br />
** Pandora is softfp and Pyra is hardfp - it's unlikely one package could work on both systems anyway.<br />
** For current OpenPandora, we can already run Debian on SD-card.<br />
*** Maybe we can provide Pyra apps for OpenPandora using the Debian-on-sd-card opposed to Debian-on-internal-space<br />
|| Very high || Decided: compatibility with SuperZaxxon in the future PND system is not important ||<br />
* http://boards.openpandora.org/topic/15530-the-software-side/page-7#entry310691<br />
* http://boards.openpandora.org/topic/15613-drop-support-for-superzaxxon-in-new-pnd-format/<br />
|-<br />
| API/Daemon to switch between joystick/keyboard mode || || || ||<br />
* Not exactly a PND-error<br />
* aTc gathered a nice list of notes: http://boards.openpandora.org/topic/15587-userspace-input-daemon/#entry311623<br />
|-<br />
| Guideline for consistent use of gaming buttons:<br />
* It should reduce the need of patches on top of the ported software<br />
||<br />
* A keyboard/joystick remapping function<br />
* A guideline for software developers<br />
|| || || No single-solution, but a set of them<br />
|-<br />
| Easier to parse than current PNDs and less fragile ||<br />
* header + metadata (UBJSON) + icon + screenshots + squashfs<br />
* tar archive<br />
** more future proof<br />
** easier for shell scripts<br />
*** with dd you can extract the tar part and so<br />
** filenames and file permissions are overhead<br />
*** But the overhead is too small to be significant?<br />
** easier to modify after-compiled<br />
*** not really a problem, because the user should be contributing to the recipe anyway<br />
*** a specialized tool can be create if another solution is chosen<br />
* zipfs uncompressed filesystem<br />
** See above sections<br />
* ar archive<br />
** Like .deb format<br />
|| || ||<br />
|-<br />
| Should the metadata include packages dependencies? ||<br />
* Yes<br />
** apt-get install package --print-uris will print download uris for dep packages. Using this uris download packages and extract all (dpkg -x 1.deb ./extracted)<br />
*** Not enough. It'll need two passes between an online and an offline system. It's impossible to be sure that an offline installation will work in one pass without including the whole repo: http://boards.openpandora.org/topic/15530-the-software-side/page-9#entry311602<br />
** EvilDragon voted for yes-and-no? http://boards.openpandora.org/topic/15530-the-software-side/page-5#entry310590<br />
*** Or does it mean a minimum "os-version required" metadata on the package and let the user update the system to have the latest libraries available?<br />
*** Possibly he doesn't care and just want something that just works: http://boards.openpandora.org/topic/15574-confliting-requirements/page-2#entry311580<br />
** http://boards.openpandora.org/topic/15530-the-software-side/page-5#entry310627<br />
* No.<br />
** EvilDragon commented that he wanna a plug-and-play-and-works system ( http://boards.openpandora.org/topic/15530-the-software-side/page-3#entry310412 )<br />
** If the user is lacking of internet access, "plug-and-play" package won't work<br />
** .deb for general-purpose software<br />
* Complete user-space like containment features (see http://www.docker.io/ for an example)<br />
|| || Decided ||<br />
* After a long heated discussion, general consensus seems to be:<br />
** Tune the makepnd script to create flat PNDs (packages which only deps are the minimum required set already present) and light PNDs (packages with .deb dependencies).<br />
*** Both packages would be included in the repo and user download what he likes more.<br />
|-<br />
| Minimum set of dependencies (SDL, ...) || || || WIP || Progress is happenning on github:<br />
* https://github.com/Pyra-Handheld/TODO/issues/5<br />
* https://github.com/Pyra-Handheld/TODO/issues/7<br />
|-<br />
| It should be click-and-run ||<br />
* binfmt<br />
* Sheebang<br />
* File (extension) association<br />
|| || ||<br />
|-<br />
| Less duplication of metadata (c.f. PXML files) || || || ||<br />
|-<br />
| AUR-like system:<br />
* Building recipes is easier to replicate and contribute<br />
* If maintainer is jerk, moderator can move his maintanership<br />
|| Fork makepkg || || [http://boards.openpandora.org/topic/15530-the-software-side/page-7#entry310691 Cloudef is working on this task] ||<br />
|-<br />
| A compatibility layer for the current *.PND system:<br />
* The more transparent, the better<br />
||<br />
* Dual-boot<br />
* Virtual Machine<br />
* Tune Debian system to include all libraries used in current SuperZaxxon<br />
** OpenEmbedded it's based on the Debian, afterall<br />
** Probably won't happen because of hard/soft float point library issues<br />
* chroot<br />
** chroot doesn't start services like the real system, then the services of the chroot must be replaced/configured<br />
|| High || ||<br />
* Pyra will have larger internal space, then duplicate system isn't a large problem<br />
* The compatibility layer should have its own table?<br />
* https://github.com/Pyra-Handheld/TODO/issues/2<br />
|-<br />
| Cross-building? || || || Undecided ||<br />
|-<br />
| Home-dir of applications ||<br />
* Maybe one per SD-card<br />
* Union-fs? Aufs? [http://pdos.csail.mit.edu/mbox/ Mbox]?<br />
* overridable by env var?<br />
|| || ||<br />
|-<br />
| Overridable scripts ||<br />
* By creating files with the right name, you can override files in the package (e.g. scripts)<br />
* Yes:<br />
** Cool for hackers<br />
* No:<br />
** Necessitates a union filesystem<br />
** Not absolutely necessary - hackers can just rebuild the package.<br />
|| || ||<br />
|-<br />
| File-extension? ||<br />
* .pyra<br />
** We're not using fucking DOS for lord-sake, what's the point of use 3-letters extension?<br />
* .pyd for pyra-data<br />
* .pyr<br />
* .pp for pyra program<br />
* .pa for pyra app<br />
* .drp for dragonbox pyra<br />
** Why limit this to just the Pyra? Could be useful in future systems/elsewhere too.<br />
* ...<br />
|| || undecided. || Maybe a series of polls on the boards? ;)<br />
|-<br />
| Keep what is good || Document all of the good features and port the content to this table || High || Not started ||<br />
|-<br />
| Extra bonus points for a design that can easily be used on other distros and devices as well || || Optional || Won't fix ||<br />
* .deb dependencies already kills the system-agnostic system<br />
** Unless we submit our changes to upstream Debian, which is unlikely to happen<br />
|-<br />
| Multiple binaries/icons per app || || || || Somebody (lost on the sea of comments) didn't like this design, but it should be possible for special software<br />
|-<br />
| Delta packages || || || || http://boards.openpandora.org/topic/15530-the-software-side/page-9#entry311600<br />
|-<br />
| PATH handling || || || || http://boards.openpandora.org/topic/15530-the-software-side/page-9#entry311618<br />
|}<br />
<br />
[[Category:PND]]<br />
[[Category:Ideas and proposals]]</div>Vinipsmakerhttps://pandorawiki.org/index.php?title=New_PND_format&diff=29493New PND format2014-02-09T20:41:43Z<p>Vinipsmaker: </p>
<hr />
<div>The current PND format has some shortcomings as listed below. This page should serve as a discussion page/white board for how the format could be improved.<br />
<br />
''' Proposal for discussion, this is just some opening shots '''<br />
<br />
== Current situation ==<br />
The current (ISO-based) PND format has the following shortcomings:<br />
<br />
* It currently uses ISO, SquashFS, and other filesystems... This means that there's no standard to follow, and libpnd needs to carry around support for a multitude of file systems.<br />
* It (often, but not always) uses the [http://en.wikipedia.org/wiki/ISO_9660 ISO file system] which is inefficient at storing data, because:<br />
** It contains duplicate headers for each file entry[http://users.telenet.be/it3.consultants.bvba/handouts/ISO9960.html] (one version with big-endian integers, one version with little-endian integers[http://en.wikipedia.org/wiki/Endianness]) which leads to many kilobytes of wasted space.<br />
** Its file tables are fixed-size, and there are therefore limitations on how many folders you can have, what names you can give your files, how big your files can be etc. For instance, files can only have names that are max 31 characters long, all upper-case, and limited to the ASCII character encoding, and an ISO does only support a folder depth of 8[http://www.on-time.com/rtos-32-docs/rtfiles-32/programming-manual/iso-9660/]. To support more, the Joliet file system extension is needed (or isofs won't recognize all file paths), and the Joliet header can be placed practically anywhere in the file, which means additional seek times.<br />
** It needs various file system extensions to behave correctly, like for example [http://en.wikipedia.org/wiki/Rock_Ridge Rock Ridge Interchange Protocol] and [http://en.wikipedia.org/wiki/Joliet_(file_system) Joliet]. Without them, it becomes pretty useless as demonstrated above, and with them, it becomes difficult to read by a tool (You can only use isofs! There are no other tools out there for programmers to use!).<br />
** It's very difficult to use since you need special ISO making tools to create the image. ISO is a relatively rare format if you're not technically inclined.<br />
** If you want to add or remove files from it, it's near impossible if you use a compact ISO file, since there's no way that you can "expand" an existing ISO file.<br />
* The PND header data is at the end (or in the middle of the file if a screenshot is included) which makes it impossible for e.g. libmagic to recognize the PND file. It will instead recognize it as an ISO file. Having the header data at the end also makes it take a very long time to find the data, making tools and the libpnd library very inefficient.<br />
* The PND file uses a custom XML format for its metadata. There's no reason to do this, especially since the established ".desktop" file format fills exactly the same function.<br />
(note: ".desktop" does not contain all metadata we have wanted over time, but it's of course possible to add extensions à la "X-Pandora-Whatever=xyz")<br />
* There's no "index" for the PND file. The whole file has to be scanned (albeit backwards) to find a PXML file, and there's a big chance for false positives etc.<br />
* Data is just appended linearly to the file so there's no order. If the format is to be extended (to e.g. include an icon file after the screenshot file), should the data just be appended as well?<br />
* UTF-8 is strictly the only encoding that is supported. If you make your PXML on a Windows machine, it won't work.<br />
<br />
== Proposed revisions ==<br />
=== Step 1: File system ===<br />
The file system for PNDs should be replaced by the ''uncompressed'' ZIP archive format. ZIP has the advantage that it's incredibly compact, and uncompressed ZIP makes it possible to read data from the file without having to do any decompression.<br />
<br />
ZIP files are mountable using various implementations of zipfs, and most of these implementations won't store files in memory when they are being read if the ZIP file is uncompressed.<br />
<br />
=== Step 2: Metadata ===<br />
PND files should no longer require special tools for them to be created. Therefore, since ZIP files support random access on files, it shouldn't be necessary to append/prepend metadata to the file. The user simply includes the "PXML.xml" and "preview.png" files inside of the ZIP, and any tools that need information about the PND can simply go to the central directory of the ZIP file (or use a simple ZIP library to do it for them) and get the location of the file inside of the ZIP. This should also dramatically decrease loading times for PNDs.<br />
<br />
Note; performance testing is needed; at the time we started, zip was enormously slower than plain ISO; with driver changes, it may or may not be so, hence our adopted multiple-filesystem-type system. zipfs is one possible option.<br />
<br />
=== Step 3: Metadata format ===<br />
Current PND files include so-called "PXML.xml" files. These files have a custom XML format that has a strange structure.<br />
<br />
These files should be replaced by ".desktop" files. A PND can contain one or more ".desktop" files in its root directory that specify how an application should be launched. PND tools simply use all ".desktop" files they can find in the PND when creating launchers for the contents of the PND.<br />
<br />
== Advantages ==<br />
* There don't have to be ''any'' special tools for reading PND files. The package can be run on any platform using any programming language that can read ZIP files.<br />
* We can use existing facilities to manage the launching of applications. The ".desktop" files can basically be copied without modification into standard locations of the system, and all launchers will become aware of them.<br />
* Reading PND files becomes easier (because of better tool support... sorry, but you can't link to libpnd in all programming languages) and quicker (becuase of the central directory for random file access).<br />
<br />
=== Benchmarks ===<br />
A benchmark was carried through to measure the performance of various package implementations.<br />
The compared systems were:<br />
* ZIP-based uncompressed packages using fuse-zip (a zipfs implementation) to read the files<br />
* ZIP-based compressed packages also using fuse-zip.<br />
* ISO8859-based packages using isofs<br />
* CramFS-based packages.<br />
<br />
A total of 9 files were generated for the test. They have sizes 1, 2, 3, 4, 16, 32, 64, 128 and 256 MB, and were generated via "dd if=/dev/urandom of=$file bs=1M count=$size".<br />
<br />
Because the files contain random data, they did not compress very well. The resulting files were:<br />
* "zipcompressed.zip" 511.1 MB<br />
* "zipuncompressed.zip" 511.0 MB<br />
* "iso.iso" 511.3 MB<br />
* "cramfs.image" 95.4 MB (WOW!)<br />
<br />
Some notes:<br />
* No write tests were done for obvious reasons<br />
* No random access tests were done since the Pandora will use SD cards and thus not get penalized from random access, and that's not what we're interested in anyways.<br />
* Before each test, the commands "sync; echo 3 > /proc/sys/vm/drop_caches" were run.<br />
<br />
==== First test: linear read time ====<br />
Command: "time cat mountpoint/* > /dev/null"<br />
<br />
* ISO: 8.119 seconds<br />
* CramFS: 1.488 seconds (WOW!)<br />
* Zip (compressed): 8.535 seconds<br />
* Zip (uncompressed): 8.290 seconds<br />
<br />
==== Second test: RAM usage ====<br />
-- TODO, have to find a good method of measuring this --<br />
<br />
== Remaining issues ==<br />
* How should libpnd tell the difference between old and new PND files? Should it depend on the "file" tool, should it run its own recognition algorithm, or what? Should the extension be changed to e.g. ".box" (Which was proposed in a thread)?<br />
* How to make sure that the ZIP files are uncompressed? Should we provide a script that "uncompresses" an ordinary ZIP file?<br />
* Is cramfs better than ZIP?<br />
** cramfs cannot be uncompressed like ZIP can<br />
** If you compare compressed ZIP and cramfs, cramfs is more efficient.<br />
** cramfs is harder to use than ZIP (for the developer).<br />
** cramfs ''needs'' to have all of its opened files in memory.<br />
<br />
== Upgrade path from the old PND format ==<br />
A simple script can be written that extracts the old PND file, including the screenshot and the PXML.xml file. The PXML.xml file is then converted to one or many .desktop files, and the desktop files, the preview.png file, and the package contents are compacted into a ZIP that then is renamed to "*.pnd"<br />
<br />
== Usage scenario ==<br />
=== Repackaging of an application from another package format ===<br />
* The user grabs the package for the application<br />
* He dumps the executable and all required libraries into a folder<br />
* He dumps a "screenshot.png" file into the folder that he's made<br />
* He copies the ".desktop" file(s) for the application from the old package, opens them, replaces "Exec=/usr/bin/bla" with "Exec=./bla" and saves them in the directory<br />
* He uses a ZIP archiver to make a ZIP out of the folder, making sure that he sets the "uncompressed" option.<br />
<br />
=== Creating PNDs as part of a build process ===<br />
* The build tool creates a directory with all of the necessary information like above and invokes the "zip" utility to compress the folder.<br />
<br />
=== Accessing a PND's contents for the... ===<br />
<br />
==== ...lazy programmer ====<br />
* The programmer extracts the PND into a folder via libzip and accesses its contents.<br />
<br />
==== ...pragmatic programmer ====<br />
* The programmer uses zipfs to mount the ZIP and accesses the file's contents.<br />
<br />
==== ...smart performance-aware programmer ====<br />
* The programmer fseek()'s the ZIP file until he finds 0x04034b50.<br />
* He jumps 18 bytes forward and reads the int at that location, storing it in "length".<br />
* He jumps 4 bytes and checks if this int matches "length". If it doesn't, it means that the file is compressed, and an error is reported.<br />
* He then jumps forward 4 bytes and stores the short at that location in a variable "nameLength".<br />
* Another 2-byte jump gets the short "extensionsLength".<br />
* He then jumps 2 bytes and reads "nameLength" amount of bytes from the file.<br />
* He then uses strcmp() to see if this string matches the sought-after file. If not, he continues the fseek().<br />
* The programmer now skips "entensionsLength" amount of bytes.<br />
* The programmer reads "length" amount of bytes from the file, and uses this as the sought-after file data.<br />
<br />
==== ...performance fascist/programmer who wants low seek times ====<br />
* The programmer fseek()'s the file from the end until he finds 0x02014b50.<br />
* He then uses the following table to get the information he needs:<br />
{|class="wikitable"<br />
|+ ZIP central directory file header<br />
|-<br />
! Offset !! Bytes !! Description<br />
|-<br />
| 0 || 4 || Central directory file header signature = 0x02014b50<br />
|-<br />
| 4 || 2 || Version made by<br />
|-<br />
| 6 || 2 || Version needed to extract (minimum)<br />
|-<br />
| 8 || 2 || General purpose bit flag<br />
|-<br />
| 10 || 2 || Compression method<br />
|-<br />
| 12 || 2 || File last modification time<br />
|-<br />
| 14 || 2 || File last modification date<br />
|-<br />
| 16 || 4 || CRC-32<br />
|-<br />
| 20 || 4 || Compressed size<br />
|-<br />
| 24 || 4 || Uncompressed size<br />
|-<br />
| 28 || 2 || File name length (''n'')<br />
|-<br />
| 30 || 2 || Extra field length (''m'')<br />
|-<br />
| 32 || 2 || File comment length (''k'')<br />
|-<br />
| 34 || 2 || Disk number where file starts<br />
|-<br />
| 36 || 2 || Internal file attributes<br />
|-<br />
| 38 || 4 || External file attributes<br />
|-<br />
| 42 || 4 || Relative offset of local file header<br />
|-<br />
| 46 || ''n'' || File name<br />
|-<br />
| 46+''n'' || ''m'' || Extra field<br />
|-<br />
| 46+''n''+''m'' || ''k'' || File comment<br />
|}<br />
<br />
* The relative file offset can then be used to jump to the file in question. The file is then scanned as described in the previous section.<br />
<br />
== The 2014/Pyra effort ==<br />
<br />
A new format is being developed for the DragonBox Pyra handheld. Here are the requirements/notes:<br />
<br />
{| class="wikitable"<br />
|-<br />
! Requirement !! Solutions !! Importance !! Status (decided, undecided, post-proned, won't fix, ...) !! Notes<br />
|-<br />
| Should the new format be supported by Pandora AND Pyra? ||<br />
* Yes<br />
** Makes releasing compatible applications easier.<br />
* no<br />
** Pandora is softfp and Pyra is hardfp - it's unlikely one package could work on both systems anyway.<br />
** For current OpenPandora, we can already run Debian on SD-card.<br />
*** Maybe we can provide Pyra apps for OpenPandora using the Debian-on-sd-card opposed to Debian-on-internal-space<br />
|| Very high || Decided: compatibility with SuperZaxxon in the future PND system is not important ||<br />
* http://boards.openpandora.org/topic/15530-the-software-side/page-7#entry310691<br />
* http://boards.openpandora.org/topic/15613-drop-support-for-superzaxxon-in-new-pnd-format/<br />
|-<br />
| API/Daemon to switch between joystick/keyboard mode || || || ||<br />
* Not exactly a PND-error<br />
* aTc gathered a nice list of notes: http://boards.openpandora.org/topic/15587-userspace-input-daemon/#entry311623<br />
|-<br />
| Guideline for consistent use of gaming buttons:<br />
* It should reduce the need of patches on top of the ported software<br />
||<br />
* A keyboard/joystick remapping function<br />
* A guideline for software developers<br />
|| || || No single-solution, but a set of them<br />
|-<br />
| Easier to parse than current PNDs and less fragile ||<br />
* header + metadata (UBJSON) + icon + screenshots + squashfs<br />
* tar archive<br />
** more future proof<br />
** easier for shell scripts<br />
*** with dd you can extract the tar part and so<br />
** filenames and file permissions are overhead<br />
*** But the overhead is too small to be significant?<br />
** easier to modify after-compiled<br />
*** not really a problem, because the user should be contributing to the recipe anyway<br />
*** a specialized tool can be create if another solution is chosen<br />
* zipfs uncompressed filesystem<br />
** See above sections<br />
* ar archive<br />
** Like .deb format<br />
|| || ||<br />
|-<br />
| Should the metadata include packages dependencies? ||<br />
* Yes<br />
** apt-get install package --print-uris will print download uris for dep packages. Using this uris download packages and extract all (dpkg -x 1.deb ./extracted)<br />
*** Not enough. It'll need two passes between an online and an offline system. It's impossible to be sure that an offline installation will work in one pass without including the whole repo: http://boards.openpandora.org/topic/15530-the-software-side/page-9#entry311602<br />
** EvilDragon voted for yes-and-no? http://boards.openpandora.org/topic/15530-the-software-side/page-5#entry310590<br />
*** Or does it mean a minimum "os-version required" metadata on the package and let the user update the system to have the latest libraries available?<br />
*** Possibly he doesn't care and just want something that just works: http://boards.openpandora.org/topic/15574-confliting-requirements/page-2#entry311580<br />
** http://boards.openpandora.org/topic/15530-the-software-side/page-5#entry310627<br />
* No.<br />
** EvilDragon commented that he wanna a plug-and-play-and-works system ( http://boards.openpandora.org/topic/15530-the-software-side/page-3#entry310412 )<br />
** If the user is lacking of internet access, "plug-and-play" package won't work<br />
** .deb for general-purpose software<br />
* Complete user-space like containment features (see http://www.docker.io/ for an example)<br />
|| || Decided ||<br />
* After a long heated discussion, general consensus seems to be:<br />
** Tune the makepnd script to create flat PNDs (packages which only deps are the minimum required set already present) and light PNDs (packages with .deb dependencies).<br />
*** Both packages would be included in the repo and user download what he likes more.<br />
|-<br />
| Minimum set of dependencies (SDL, ...) || || || WIP || Progress is happenning on github:<br />
* https://github.com/Pyra-Handheld/TODO/issues/5<br />
* https://github.com/Pyra-Handheld/TODO/issues/7<br />
|-<br />
| It should be click-and-run ||<br />
* binfmt<br />
* Sheebang<br />
* File (extension) association<br />
|| || ||<br />
|-<br />
| Less duplication of metadata (c.f. PXML files) || || || ||<br />
|-<br />
| AUR-like system:<br />
* Building recipes is easier to replicate and contribute<br />
* If maintainer is jerk, moderator can move his maintanership<br />
|| Fork makepkg || || [http://boards.openpandora.org/topic/15530-the-software-side/page-7#entry310691 Cloudef is working on this task] ||<br />
|-<br />
| A compatibility layer for the current *.PND system:<br />
* The more transparent, the better<br />
||<br />
* Dual-boot<br />
* Virtual Machine<br />
* Tune Debian system to include all libraries used in current SuperZaxxon<br />
** OpenEmbedded it's based on the Debian, afterall<br />
** Probably won't happen because of hard/soft float point library issues<br />
* chroot<br />
** chroot doesn't start services like the real system, then the services of the chroot must be replaced/configured<br />
|| High || ||<br />
* Pyra will have larger internal space, then duplicate system isn't a large problem<br />
* The compatibility layer should have its own table?<br />
* https://github.com/Pyra-Handheld/TODO/issues/2<br />
|-<br />
| Cross-building? || || || Undecided ||<br />
|-<br />
| Home-dir of applications ||<br />
* Maybe one per SD-card<br />
* Union-fs? Aufs?<br />
* overridable by env var?<br />
|| || ||<br />
|-<br />
| Overridable scripts ||<br />
* By creating files with the right name, you can override files in the package (e.g. scripts)<br />
* Yes:<br />
** Cool for hackers<br />
* No:<br />
** Necessitates a union filesystem<br />
** Not absolutely necessary - hackers can just rebuild the package.<br />
|| || ||<br />
|-<br />
| File-extension? ||<br />
* .pyra<br />
** We're not using fucking DOS for lord-sake, what's the point of use 3-letters extension?<br />
* .pyd for pyra-data<br />
* .pyr<br />
* .pp for pyra program<br />
* .pa for pyra app<br />
* .drp for dragonbox pyra<br />
** Why limit this to just the Pyra? Could be useful in future systems/elsewhere too.<br />
* ...<br />
|| || undecided. || Maybe a series of polls on the boards? ;)<br />
|-<br />
| Keep what is good || Document all of the good features and port the content to this table || High || Not started ||<br />
|-<br />
| Extra bonus points for a design that can easily be used on other distros and devices as well || || Optional || Won't fix ||<br />
* .deb dependencies already kills the system-agnostic system<br />
** Unless we submit our changes to upstream Debian, which is unlikely to happen<br />
|-<br />
| Multiple binaries/icons per app || || || || Somebody (lost on the sea of comments) didn't like this design, but it should be possible for special software<br />
|-<br />
| Delta packages || || || || http://boards.openpandora.org/topic/15530-the-software-side/page-9#entry311600<br />
|-<br />
| PATH handling || || || || http://boards.openpandora.org/topic/15530-the-software-side/page-9#entry311618<br />
|}<br />
<br />
[[Category:PND]]<br />
[[Category:Ideas and proposals]]</div>Vinipsmakerhttps://pandorawiki.org/index.php?title=New_PND_format&diff=29492New PND format2014-02-09T20:39:13Z<p>Vinipsmaker: </p>
<hr />
<div>The current PND format has some shortcomings as listed below. This page should serve as a discussion page/white board for how the format could be improved.<br />
<br />
''' Proposal for discussion, this is just some opening shots '''<br />
<br />
== Current situation ==<br />
The current (ISO-based) PND format has the following shortcomings:<br />
<br />
* It currently uses ISO, SquashFS, and other filesystems... This means that there's no standard to follow, and libpnd needs to carry around support for a multitude of file systems.<br />
* It (often, but not always) uses the [http://en.wikipedia.org/wiki/ISO_9660 ISO file system] which is inefficient at storing data, because:<br />
** It contains duplicate headers for each file entry[http://users.telenet.be/it3.consultants.bvba/handouts/ISO9960.html] (one version with big-endian integers, one version with little-endian integers[http://en.wikipedia.org/wiki/Endianness]) which leads to many kilobytes of wasted space.<br />
** Its file tables are fixed-size, and there are therefore limitations on how many folders you can have, what names you can give your files, how big your files can be etc. For instance, files can only have names that are max 31 characters long, all upper-case, and limited to the ASCII character encoding, and an ISO does only support a folder depth of 8[http://www.on-time.com/rtos-32-docs/rtfiles-32/programming-manual/iso-9660/]. To support more, the Joliet file system extension is needed (or isofs won't recognize all file paths), and the Joliet header can be placed practically anywhere in the file, which means additional seek times.<br />
** It needs various file system extensions to behave correctly, like for example [http://en.wikipedia.org/wiki/Rock_Ridge Rock Ridge Interchange Protocol] and [http://en.wikipedia.org/wiki/Joliet_(file_system) Joliet]. Without them, it becomes pretty useless as demonstrated above, and with them, it becomes difficult to read by a tool (You can only use isofs! There are no other tools out there for programmers to use!).<br />
** It's very difficult to use since you need special ISO making tools to create the image. ISO is a relatively rare format if you're not technically inclined.<br />
** If you want to add or remove files from it, it's near impossible if you use a compact ISO file, since there's no way that you can "expand" an existing ISO file.<br />
* The PND header data is at the end (or in the middle of the file if a screenshot is included) which makes it impossible for e.g. libmagic to recognize the PND file. It will instead recognize it as an ISO file. Having the header data at the end also makes it take a very long time to find the data, making tools and the libpnd library very inefficient.<br />
* The PND file uses a custom XML format for its metadata. There's no reason to do this, especially since the established ".desktop" file format fills exactly the same function.<br />
(note: ".desktop" does not contain all metadata we have wanted over time, but it's of course possible to add extensions à la "X-Pandora-Whatever=xyz")<br />
* There's no "index" for the PND file. The whole file has to be scanned (albeit backwards) to find a PXML file, and there's a big chance for false positives etc.<br />
* Data is just appended linearly to the file so there's no order. If the format is to be extended (to e.g. include an icon file after the screenshot file), should the data just be appended as well?<br />
* UTF-8 is strictly the only encoding that is supported. If you make your PXML on a Windows machine, it won't work.<br />
<br />
== Proposed revisions ==<br />
=== Step 1: File system ===<br />
The file system for PNDs should be replaced by the ''uncompressed'' ZIP archive format. ZIP has the advantage that it's incredibly compact, and uncompressed ZIP makes it possible to read data from the file without having to do any decompression.<br />
<br />
ZIP files are mountable using various implementations of zipfs, and most of these implementations won't store files in memory when they are being read if the ZIP file is uncompressed.<br />
<br />
=== Step 2: Metadata ===<br />
PND files should no longer require special tools for them to be created. Therefore, since ZIP files support random access on files, it shouldn't be necessary to append/prepend metadata to the file. The user simply includes the "PXML.xml" and "preview.png" files inside of the ZIP, and any tools that need information about the PND can simply go to the central directory of the ZIP file (or use a simple ZIP library to do it for them) and get the location of the file inside of the ZIP. This should also dramatically decrease loading times for PNDs.<br />
<br />
Note; performance testing is needed; at the time we started, zip was enormously slower than plain ISO; with driver changes, it may or may not be so, hence our adopted multiple-filesystem-type system. zipfs is one possible option.<br />
<br />
=== Step 3: Metadata format ===<br />
Current PND files include so-called "PXML.xml" files. These files have a custom XML format that has a strange structure.<br />
<br />
These files should be replaced by ".desktop" files. A PND can contain one or more ".desktop" files in its root directory that specify how an application should be launched. PND tools simply use all ".desktop" files they can find in the PND when creating launchers for the contents of the PND.<br />
<br />
== Advantages ==<br />
* There don't have to be ''any'' special tools for reading PND files. The package can be run on any platform using any programming language that can read ZIP files.<br />
* We can use existing facilities to manage the launching of applications. The ".desktop" files can basically be copied without modification into standard locations of the system, and all launchers will become aware of them.<br />
* Reading PND files becomes easier (because of better tool support... sorry, but you can't link to libpnd in all programming languages) and quicker (becuase of the central directory for random file access).<br />
<br />
=== Benchmarks ===<br />
A benchmark was carried through to measure the performance of various package implementations.<br />
The compared systems were:<br />
* ZIP-based uncompressed packages using fuse-zip (a zipfs implementation) to read the files<br />
* ZIP-based compressed packages also using fuse-zip.<br />
* ISO8859-based packages using isofs<br />
* CramFS-based packages.<br />
<br />
A total of 9 files were generated for the test. They have sizes 1, 2, 3, 4, 16, 32, 64, 128 and 256 MB, and were generated via "dd if=/dev/urandom of=$file bs=1M count=$size".<br />
<br />
Because the files contain random data, they did not compress very well. The resulting files were:<br />
* "zipcompressed.zip" 511.1 MB<br />
* "zipuncompressed.zip" 511.0 MB<br />
* "iso.iso" 511.3 MB<br />
* "cramfs.image" 95.4 MB (WOW!)<br />
<br />
Some notes:<br />
* No write tests were done for obvious reasons<br />
* No random access tests were done since the Pandora will use SD cards and thus not get penalized from random access, and that's not what we're interested in anyways.<br />
* Before each test, the commands "sync; echo 3 > /proc/sys/vm/drop_caches" were run.<br />
<br />
==== First test: linear read time ====<br />
Command: "time cat mountpoint/* > /dev/null"<br />
<br />
* ISO: 8.119 seconds<br />
* CramFS: 1.488 seconds (WOW!)<br />
* Zip (compressed): 8.535 seconds<br />
* Zip (uncompressed): 8.290 seconds<br />
<br />
==== Second test: RAM usage ====<br />
-- TODO, have to find a good method of measuring this --<br />
<br />
== Remaining issues ==<br />
* How should libpnd tell the difference between old and new PND files? Should it depend on the "file" tool, should it run its own recognition algorithm, or what? Should the extension be changed to e.g. ".box" (Which was proposed in a thread)?<br />
* How to make sure that the ZIP files are uncompressed? Should we provide a script that "uncompresses" an ordinary ZIP file?<br />
* Is cramfs better than ZIP?<br />
** cramfs cannot be uncompressed like ZIP can<br />
** If you compare compressed ZIP and cramfs, cramfs is more efficient.<br />
** cramfs is harder to use than ZIP (for the developer).<br />
** cramfs ''needs'' to have all of its opened files in memory.<br />
<br />
== Upgrade path from the old PND format ==<br />
A simple script can be written that extracts the old PND file, including the screenshot and the PXML.xml file. The PXML.xml file is then converted to one or many .desktop files, and the desktop files, the preview.png file, and the package contents are compacted into a ZIP that then is renamed to "*.pnd"<br />
<br />
== Usage scenario ==<br />
=== Repackaging of an application from another package format ===<br />
* The user grabs the package for the application<br />
* He dumps the executable and all required libraries into a folder<br />
* He dumps a "screenshot.png" file into the folder that he's made<br />
* He copies the ".desktop" file(s) for the application from the old package, opens them, replaces "Exec=/usr/bin/bla" with "Exec=./bla" and saves them in the directory<br />
* He uses a ZIP archiver to make a ZIP out of the folder, making sure that he sets the "uncompressed" option.<br />
<br />
=== Creating PNDs as part of a build process ===<br />
* The build tool creates a directory with all of the necessary information like above and invokes the "zip" utility to compress the folder.<br />
<br />
=== Accessing a PND's contents for the... ===<br />
<br />
==== ...lazy programmer ====<br />
* The programmer extracts the PND into a folder via libzip and accesses its contents.<br />
<br />
==== ...pragmatic programmer ====<br />
* The programmer uses zipfs to mount the ZIP and accesses the file's contents.<br />
<br />
==== ...smart performance-aware programmer ====<br />
* The programmer fseek()'s the ZIP file until he finds 0x04034b50.<br />
* He jumps 18 bytes forward and reads the int at that location, storing it in "length".<br />
* He jumps 4 bytes and checks if this int matches "length". If it doesn't, it means that the file is compressed, and an error is reported.<br />
* He then jumps forward 4 bytes and stores the short at that location in a variable "nameLength".<br />
* Another 2-byte jump gets the short "extensionsLength".<br />
* He then jumps 2 bytes and reads "nameLength" amount of bytes from the file.<br />
* He then uses strcmp() to see if this string matches the sought-after file. If not, he continues the fseek().<br />
* The programmer now skips "entensionsLength" amount of bytes.<br />
* The programmer reads "length" amount of bytes from the file, and uses this as the sought-after file data.<br />
<br />
==== ...performance fascist/programmer who wants low seek times ====<br />
* The programmer fseek()'s the file from the end until he finds 0x02014b50.<br />
* He then uses the following table to get the information he needs:<br />
{|class="wikitable"<br />
|+ ZIP central directory file header<br />
|-<br />
! Offset !! Bytes !! Description<br />
|-<br />
| 0 || 4 || Central directory file header signature = 0x02014b50<br />
|-<br />
| 4 || 2 || Version made by<br />
|-<br />
| 6 || 2 || Version needed to extract (minimum)<br />
|-<br />
| 8 || 2 || General purpose bit flag<br />
|-<br />
| 10 || 2 || Compression method<br />
|-<br />
| 12 || 2 || File last modification time<br />
|-<br />
| 14 || 2 || File last modification date<br />
|-<br />
| 16 || 4 || CRC-32<br />
|-<br />
| 20 || 4 || Compressed size<br />
|-<br />
| 24 || 4 || Uncompressed size<br />
|-<br />
| 28 || 2 || File name length (''n'')<br />
|-<br />
| 30 || 2 || Extra field length (''m'')<br />
|-<br />
| 32 || 2 || File comment length (''k'')<br />
|-<br />
| 34 || 2 || Disk number where file starts<br />
|-<br />
| 36 || 2 || Internal file attributes<br />
|-<br />
| 38 || 4 || External file attributes<br />
|-<br />
| 42 || 4 || Relative offset of local file header<br />
|-<br />
| 46 || ''n'' || File name<br />
|-<br />
| 46+''n'' || ''m'' || Extra field<br />
|-<br />
| 46+''n''+''m'' || ''k'' || File comment<br />
|}<br />
<br />
* The relative file offset can then be used to jump to the file in question. The file is then scanned as described in the previous section.<br />
<br />
== The 2014/Pyra effort ==<br />
<br />
A new format is being developed for the DragonBox Pyra handheld. Here are the requirements/notes:<br />
<br />
{| class="wikitable"<br />
|-<br />
! Requirement !! Solutions !! Importance !! Status (decided, undecided, post-proned, won't fix, ...) !! Notes<br />
|-<br />
| Should the new format be supported by Pandora AND Pyra? ||<br />
* Yes<br />
** Makes releasing compatible applications easier.<br />
* no<br />
** Pandora is softfp and Pyra is hardfp - it's unlikely one package could work on both systems anyway.<br />
** For current OpenPandora, we can already run Debian on SD-card.<br />
*** Maybe we can provide Pyra apps for OpenPandora using the Debian-on-sd-card opposed to Debian-on-internal-space<br />
|| Very high || Decided: compatibility with SuperZaxxon in the future PND system is not important ||<br />
* http://boards.openpandora.org/topic/15530-the-software-side/page-7#entry310691<br />
* http://boards.openpandora.org/topic/15613-drop-support-for-superzaxxon-in-new-pnd-format/<br />
|-<br />
| API/Daemon to switch between joystick/keyboard mode || || || ||<br />
* Not exactly a PND-error<br />
* aTc gathered a nice list of notes: http://boards.openpandora.org/topic/15587-userspace-input-daemon/#entry311623<br />
|-<br />
| Guideline for consistent use of gaming buttons:<br />
* It should reduce the need of patches on top of the ported software<br />
||<br />
* A keyboard/joystick remapping function<br />
* A guideline for software developers<br />
|| || || No single-solution, but a set of them<br />
|-<br />
| Easier to parse than current PNDs and less fragile ||<br />
* header + metadata (UBJSON) + icon + screenshots + squashfs<br />
* tar archive<br />
** more future proof<br />
** easier for shell scripts<br />
*** with dd you can extract the tar part and so<br />
** filenames and file permissions are overhead<br />
*** But the overhead is too small to be significant?<br />
** easier to modify after-compiled<br />
*** not really a problem, because the user should be contributing to the recipe anyway<br />
*** a specialized tool can be create if another solution is chosen<br />
* zipfs uncompressed filesystem<br />
** See above sections<br />
* ar archive<br />
** Like .deb format<br />
|| || ||<br />
|-<br />
| Should the metadata include packages dependencies? ||<br />
* Yes<br />
** apt-get install package --print-uris will print download uris for dep packages. Using this uris download packages and extract all (dpkg -x 1.deb ./extracted)<br />
*** Not enough. It'll need two passes between an online and an offline system. It's impossible to be sure that an offline installation will work in one pass without including the whole repo: http://boards.openpandora.org/topic/15530-the-software-side/page-9#entry311602<br />
** EvilDragon voted for yes-and-no? http://boards.openpandora.org/topic/15530-the-software-side/page-5#entry310590<br />
*** Or does it mean a minimum "os-version required" metadata on the package and let the user update the system to have the latest libraries available?<br />
*** Possibly he doesn't care and just want something that just works: http://boards.openpandora.org/topic/15574-confliting-requirements/page-2#entry311580<br />
** http://boards.openpandora.org/topic/15530-the-software-side/page-5#entry310627<br />
* No.<br />
** EvilDragon commented that he wanna a plug-and-play-and-works system ( http://boards.openpandora.org/topic/15530-the-software-side/page-3#entry310412 )<br />
** If the user is lacking of internet access, "plug-and-play" package won't work<br />
** .deb for general-purpose software<br />
* Complete user-space like containment features (see http://www.docker.io/ for an example)<br />
|| || Decided ||<br />
* After a long heated discussion, general consensus seems to be:<br />
** Tune the makepnd script to create flat PNDs (packages which only deps are the minimum required set already present) and light PNDs (packages with .deb dependencies).<br />
*** Both packages would be included in the repo and user download what he likes more.<br />
|-<br />
| Minimum set of dependencies (SDL, ...) || || || WIP || Progress is happenning on github:<br />
* https://github.com/Pyra-Handheld/TODO/issues/5<br />
* https://github.com/Pyra-Handheld/TODO/issues/7<br />
|-<br />
| It should be click-and-run ||<br />
* binfmt<br />
* Sheebang<br />
* File (extension) association<br />
|| || ||<br />
|-<br />
| Less duplication of metadata (c.f. PXML files) || || || ||<br />
|-<br />
| AUR-like system:<br />
* Building recipes is easier to replicate and contribute<br />
* If maintainer is jerk, moderator can move his maintanership<br />
|| Fork makepkg || || [http://boards.openpandora.org/topic/15530-the-software-side/page-7#entry310691 Cloudef is working on this task] ||<br />
|-<br />
| A compatibility layer for the current *.PND system:<br />
* The more transparent, the better<br />
||<br />
* Dual-boot<br />
* Virtual Machine<br />
* Tune Debian system to include all libraries used in current SuperZaxxon<br />
** OpenEmbedded it's based on the Debian, afterall<br />
** Probably won't happen because of hard/soft float point library issues<br />
* chroot<br />
** chroot doesn't start services like the real system, then the services of the chroot must be replaced/configured<br />
|| High || ||<br />
* Pyra will have larger internal space, then duplicate system isn't a large problem<br />
* The compatibility layer should have its own table?<br />
|-<br />
| Cross-building? || || || Undecided ||<br />
|-<br />
| Home-dir of applications ||<br />
* Maybe one per SD-card<br />
* Union-fs? Aufs?<br />
* overridable by env var?<br />
|| || ||<br />
|-<br />
| Overridable scripts ||<br />
* By creating files with the right name, you can override files in the package (e.g. scripts)<br />
* Yes:<br />
** Cool for hackers<br />
* No:<br />
** Necessitates a union filesystem<br />
** Not absolutely necessary - hackers can just rebuild the package.<br />
|| || ||<br />
|-<br />
| File-extension? ||<br />
* .pyra<br />
** We're not using fucking DOS for lord-sake, what's the point of use 3-letters extension?<br />
* .pyd for pyra-data<br />
* .pyr<br />
* .pp for pyra program<br />
* .pa for pyra app<br />
* .drp for dragonbox pyra<br />
** Why limit this to just the Pyra? Could be useful in future systems/elsewhere too.<br />
* ...<br />
|| || undecided. || Maybe a series of polls on the boards? ;)<br />
|-<br />
| Keep what is good || Document all of the good features and port the content to this table || High || Not started ||<br />
|-<br />
| Extra bonus points for a design that can easily be used on other distros and devices as well || || Optional || Won't fix ||<br />
* .deb dependencies already kills the system-agnostic system<br />
** Unless we submit our changes to upstream Debian, which is unlikely to happen<br />
|-<br />
| Multiple binaries/icons per app || || || || Somebody (lost on the sea of comments) didn't like this design, but it should be possible for special software<br />
|-<br />
| Delta packages || || || || http://boards.openpandora.org/topic/15530-the-software-side/page-9#entry311600<br />
|-<br />
| PATH handling || || || || http://boards.openpandora.org/topic/15530-the-software-side/page-9#entry311618<br />
|}<br />
<br />
[[Category:PND]]<br />
[[Category:Ideas and proposals]]</div>Vinipsmakerhttps://pandorawiki.org/index.php?title=New_PND_format&diff=29491New PND format2014-02-09T20:37:41Z<p>Vinipsmaker: another issue solve</p>
<hr />
<div>The current PND format has some shortcomings as listed below. This page should serve as a discussion page/white board for how the format could be improved.<br />
<br />
''' Proposal for discussion, this is just some opening shots '''<br />
<br />
== Current situation ==<br />
The current (ISO-based) PND format has the following shortcomings:<br />
<br />
* It currently uses ISO, SquashFS, and other filesystems... This means that there's no standard to follow, and libpnd needs to carry around support for a multitude of file systems.<br />
* It (often, but not always) uses the [http://en.wikipedia.org/wiki/ISO_9660 ISO file system] which is inefficient at storing data, because:<br />
** It contains duplicate headers for each file entry[http://users.telenet.be/it3.consultants.bvba/handouts/ISO9960.html] (one version with big-endian integers, one version with little-endian integers[http://en.wikipedia.org/wiki/Endianness]) which leads to many kilobytes of wasted space.<br />
** Its file tables are fixed-size, and there are therefore limitations on how many folders you can have, what names you can give your files, how big your files can be etc. For instance, files can only have names that are max 31 characters long, all upper-case, and limited to the ASCII character encoding, and an ISO does only support a folder depth of 8[http://www.on-time.com/rtos-32-docs/rtfiles-32/programming-manual/iso-9660/]. To support more, the Joliet file system extension is needed (or isofs won't recognize all file paths), and the Joliet header can be placed practically anywhere in the file, which means additional seek times.<br />
** It needs various file system extensions to behave correctly, like for example [http://en.wikipedia.org/wiki/Rock_Ridge Rock Ridge Interchange Protocol] and [http://en.wikipedia.org/wiki/Joliet_(file_system) Joliet]. Without them, it becomes pretty useless as demonstrated above, and with them, it becomes difficult to read by a tool (You can only use isofs! There are no other tools out there for programmers to use!).<br />
** It's very difficult to use since you need special ISO making tools to create the image. ISO is a relatively rare format if you're not technically inclined.<br />
** If you want to add or remove files from it, it's near impossible if you use a compact ISO file, since there's no way that you can "expand" an existing ISO file.<br />
* The PND header data is at the end (or in the middle of the file if a screenshot is included) which makes it impossible for e.g. libmagic to recognize the PND file. It will instead recognize it as an ISO file. Having the header data at the end also makes it take a very long time to find the data, making tools and the libpnd library very inefficient.<br />
* The PND file uses a custom XML format for its metadata. There's no reason to do this, especially since the established ".desktop" file format fills exactly the same function.<br />
(note: ".desktop" does not contain all metadata we have wanted over time, but it's of course possible to add extensions à la "X-Pandora-Whatever=xyz")<br />
* There's no "index" for the PND file. The whole file has to be scanned (albeit backwards) to find a PXML file, and there's a big chance for false positives etc.<br />
* Data is just appended linearly to the file so there's no order. If the format is to be extended (to e.g. include an icon file after the screenshot file), should the data just be appended as well?<br />
* UTF-8 is strictly the only encoding that is supported. If you make your PXML on a Windows machine, it won't work.<br />
<br />
== Proposed revisions ==<br />
=== Step 1: File system ===<br />
The file system for PNDs should be replaced by the ''uncompressed'' ZIP archive format. ZIP has the advantage that it's incredibly compact, and uncompressed ZIP makes it possible to read data from the file without having to do any decompression.<br />
<br />
ZIP files are mountable using various implementations of zipfs, and most of these implementations won't store files in memory when they are being read if the ZIP file is uncompressed.<br />
<br />
=== Step 2: Metadata ===<br />
PND files should no longer require special tools for them to be created. Therefore, since ZIP files support random access on files, it shouldn't be necessary to append/prepend metadata to the file. The user simply includes the "PXML.xml" and "preview.png" files inside of the ZIP, and any tools that need information about the PND can simply go to the central directory of the ZIP file (or use a simple ZIP library to do it for them) and get the location of the file inside of the ZIP. This should also dramatically decrease loading times for PNDs.<br />
<br />
Note; performance testing is needed; at the time we started, zip was enormously slower than plain ISO; with driver changes, it may or may not be so, hence our adopted multiple-filesystem-type system. zipfs is one possible option.<br />
<br />
=== Step 3: Metadata format ===<br />
Current PND files include so-called "PXML.xml" files. These files have a custom XML format that has a strange structure.<br />
<br />
These files should be replaced by ".desktop" files. A PND can contain one or more ".desktop" files in its root directory that specify how an application should be launched. PND tools simply use all ".desktop" files they can find in the PND when creating launchers for the contents of the PND.<br />
<br />
== Advantages ==<br />
* There don't have to be ''any'' special tools for reading PND files. The package can be run on any platform using any programming language that can read ZIP files.<br />
* We can use existing facilities to manage the launching of applications. The ".desktop" files can basically be copied without modification into standard locations of the system, and all launchers will become aware of them.<br />
* Reading PND files becomes easier (because of better tool support... sorry, but you can't link to libpnd in all programming languages) and quicker (becuase of the central directory for random file access).<br />
<br />
=== Benchmarks ===<br />
A benchmark was carried through to measure the performance of various package implementations.<br />
The compared systems were:<br />
* ZIP-based uncompressed packages using fuse-zip (a zipfs implementation) to read the files<br />
* ZIP-based compressed packages also using fuse-zip.<br />
* ISO8859-based packages using isofs<br />
* CramFS-based packages.<br />
<br />
A total of 9 files were generated for the test. They have sizes 1, 2, 3, 4, 16, 32, 64, 128 and 256 MB, and were generated via "dd if=/dev/urandom of=$file bs=1M count=$size".<br />
<br />
Because the files contain random data, they did not compress very well. The resulting files were:<br />
* "zipcompressed.zip" 511.1 MB<br />
* "zipuncompressed.zip" 511.0 MB<br />
* "iso.iso" 511.3 MB<br />
* "cramfs.image" 95.4 MB (WOW!)<br />
<br />
Some notes:<br />
* No write tests were done for obvious reasons<br />
* No random access tests were done since the Pandora will use SD cards and thus not get penalized from random access, and that's not what we're interested in anyways.<br />
* Before each test, the commands "sync; echo 3 > /proc/sys/vm/drop_caches" were run.<br />
<br />
==== First test: linear read time ====<br />
Command: "time cat mountpoint/* > /dev/null"<br />
<br />
* ISO: 8.119 seconds<br />
* CramFS: 1.488 seconds (WOW!)<br />
* Zip (compressed): 8.535 seconds<br />
* Zip (uncompressed): 8.290 seconds<br />
<br />
==== Second test: RAM usage ====<br />
-- TODO, have to find a good method of measuring this --<br />
<br />
== Remaining issues ==<br />
* How should libpnd tell the difference between old and new PND files? Should it depend on the "file" tool, should it run its own recognition algorithm, or what? Should the extension be changed to e.g. ".box" (Which was proposed in a thread)?<br />
* How to make sure that the ZIP files are uncompressed? Should we provide a script that "uncompresses" an ordinary ZIP file?<br />
* Is cramfs better than ZIP?<br />
** cramfs cannot be uncompressed like ZIP can<br />
** If you compare compressed ZIP and cramfs, cramfs is more efficient.<br />
** cramfs is harder to use than ZIP (for the developer).<br />
** cramfs ''needs'' to have all of its opened files in memory.<br />
<br />
== Upgrade path from the old PND format ==<br />
A simple script can be written that extracts the old PND file, including the screenshot and the PXML.xml file. The PXML.xml file is then converted to one or many .desktop files, and the desktop files, the preview.png file, and the package contents are compacted into a ZIP that then is renamed to "*.pnd"<br />
<br />
== Usage scenario ==<br />
=== Repackaging of an application from another package format ===<br />
* The user grabs the package for the application<br />
* He dumps the executable and all required libraries into a folder<br />
* He dumps a "screenshot.png" file into the folder that he's made<br />
* He copies the ".desktop" file(s) for the application from the old package, opens them, replaces "Exec=/usr/bin/bla" with "Exec=./bla" and saves them in the directory<br />
* He uses a ZIP archiver to make a ZIP out of the folder, making sure that he sets the "uncompressed" option.<br />
<br />
=== Creating PNDs as part of a build process ===<br />
* The build tool creates a directory with all of the necessary information like above and invokes the "zip" utility to compress the folder.<br />
<br />
=== Accessing a PND's contents for the... ===<br />
<br />
==== ...lazy programmer ====<br />
* The programmer extracts the PND into a folder via libzip and accesses its contents.<br />
<br />
==== ...pragmatic programmer ====<br />
* The programmer uses zipfs to mount the ZIP and accesses the file's contents.<br />
<br />
==== ...smart performance-aware programmer ====<br />
* The programmer fseek()'s the ZIP file until he finds 0x04034b50.<br />
* He jumps 18 bytes forward and reads the int at that location, storing it in "length".<br />
* He jumps 4 bytes and checks if this int matches "length". If it doesn't, it means that the file is compressed, and an error is reported.<br />
* He then jumps forward 4 bytes and stores the short at that location in a variable "nameLength".<br />
* Another 2-byte jump gets the short "extensionsLength".<br />
* He then jumps 2 bytes and reads "nameLength" amount of bytes from the file.<br />
* He then uses strcmp() to see if this string matches the sought-after file. If not, he continues the fseek().<br />
* The programmer now skips "entensionsLength" amount of bytes.<br />
* The programmer reads "length" amount of bytes from the file, and uses this as the sought-after file data.<br />
<br />
==== ...performance fascist/programmer who wants low seek times ====<br />
* The programmer fseek()'s the file from the end until he finds 0x02014b50.<br />
* He then uses the following table to get the information he needs:<br />
{|class="wikitable"<br />
|+ ZIP central directory file header<br />
|-<br />
! Offset !! Bytes !! Description<br />
|-<br />
| 0 || 4 || Central directory file header signature = 0x02014b50<br />
|-<br />
| 4 || 2 || Version made by<br />
|-<br />
| 6 || 2 || Version needed to extract (minimum)<br />
|-<br />
| 8 || 2 || General purpose bit flag<br />
|-<br />
| 10 || 2 || Compression method<br />
|-<br />
| 12 || 2 || File last modification time<br />
|-<br />
| 14 || 2 || File last modification date<br />
|-<br />
| 16 || 4 || CRC-32<br />
|-<br />
| 20 || 4 || Compressed size<br />
|-<br />
| 24 || 4 || Uncompressed size<br />
|-<br />
| 28 || 2 || File name length (''n'')<br />
|-<br />
| 30 || 2 || Extra field length (''m'')<br />
|-<br />
| 32 || 2 || File comment length (''k'')<br />
|-<br />
| 34 || 2 || Disk number where file starts<br />
|-<br />
| 36 || 2 || Internal file attributes<br />
|-<br />
| 38 || 4 || External file attributes<br />
|-<br />
| 42 || 4 || Relative offset of local file header<br />
|-<br />
| 46 || ''n'' || File name<br />
|-<br />
| 46+''n'' || ''m'' || Extra field<br />
|-<br />
| 46+''n''+''m'' || ''k'' || File comment<br />
|}<br />
<br />
* The relative file offset can then be used to jump to the file in question. The file is then scanned as described in the previous section.<br />
<br />
== The 2014/Pyra effort ==<br />
<br />
A new format is being developed for the DragonBox Pyra handheld. Here are the requirements/notes:<br />
<br />
{| class="wikitable"<br />
|-<br />
! Requirement !! Solutions !! Importance !! Status (decided, undecided, post-proned, won't fix, ...) !! Notes<br />
|-<br />
| Should the new format be supported by Pandora AND Pyra? ||<br />
* Yes<br />
** Makes releasing compatible applications easier.<br />
* no<br />
** Pandora is softfp and Pyra is hardfp - it's unlikely one package could work on both systems anyway.<br />
** For current OpenPandora, we can already run Debian on SD-card.<br />
*** Maybe we can provide Pyra apps for OpenPandora using the Debian-on-sd-card opposed to Debian-on-internal-space<br />
|| Very high || Decided: compatibility with SuperZaxxon in the future PND system is not important || http://boards.openpandora.org/topic/15613-drop-support-for-superzaxxon-in-new-pnd-format/<br />
* http://boards.openpandora.org/topic/15530-the-software-side/page-7#entry310691<br />
* This decision affects several of the requirements and possible solutions on this table and should be decided sooner<br />
|-<br />
| API/Daemon to switch between joystick/keyboard mode || || || ||<br />
* Not exactly a PND-error<br />
* aTc gathered a nice list of notes: http://boards.openpandora.org/topic/15587-userspace-input-daemon/#entry311623<br />
|-<br />
| Guideline for consistent use of gaming buttons:<br />
* It should reduce the need of patches on top of the ported software<br />
||<br />
* A keyboard/joystick remapping function<br />
* A guideline for software developers<br />
|| || || No single-solution, but a set of them<br />
|-<br />
| Easier to parse than current PNDs and less fragile ||<br />
* header + metadata (UBJSON) + icon + screenshots + squashfs<br />
* tar archive<br />
** more future proof<br />
** easier for shell scripts<br />
*** with dd you can extract the tar part and so<br />
** filenames and file permissions are overhead<br />
*** But the overhead is too small to be significant?<br />
** easier to modify after-compiled<br />
*** not really a problem, because the user should be contributing to the recipe anyway<br />
*** a specialized tool can be create if another solution is chosen<br />
* zipfs uncompressed filesystem<br />
** See above sections<br />
* ar archive<br />
** Like .deb format<br />
|| || ||<br />
|-<br />
| Should the metadata include packages dependencies? ||<br />
* Yes<br />
** apt-get install package --print-uris will print download uris for dep packages. Using this uris download packages and extract all (dpkg -x 1.deb ./extracted)<br />
*** Not enough. It'll need two passes between an online and an offline system. It's impossible to be sure that an offline installation will work in one pass without including the whole repo: http://boards.openpandora.org/topic/15530-the-software-side/page-9#entry311602<br />
** EvilDragon voted for yes-and-no? http://boards.openpandora.org/topic/15530-the-software-side/page-5#entry310590<br />
*** Or does it mean a minimum "os-version required" metadata on the package and let the user update the system to have the latest libraries available?<br />
*** Possibly he doesn't care and just want something that just works: http://boards.openpandora.org/topic/15574-confliting-requirements/page-2#entry311580<br />
** http://boards.openpandora.org/topic/15530-the-software-side/page-5#entry310627<br />
* No.<br />
** EvilDragon commented that he wanna a plug-and-play-and-works system ( http://boards.openpandora.org/topic/15530-the-software-side/page-3#entry310412 )<br />
** If the user is lacking of internet access, "plug-and-play" package won't work<br />
** .deb for general-purpose software<br />
* Complete user-space like containment features (see http://www.docker.io/ for an example)<br />
|| || Decided ||<br />
* After a long heated discussion, general consensus seems to be:<br />
** Tune the makepnd script to create flat PNDs (packages which only deps are the minimum required set already present) and light PNDs (packages with .deb dependencies).<br />
*** Both packages would be included in the repo and user download what he likes more.<br />
|-<br />
| Minimum set of dependencies (SDL, ...) || || || WIP || Progress is happenning on github:<br />
* https://github.com/Pyra-Handheld/TODO/issues/5<br />
* https://github.com/Pyra-Handheld/TODO/issues/7<br />
|-<br />
| It should be click-and-run ||<br />
* binfmt<br />
* Sheebang<br />
* File (extension) association<br />
|| || ||<br />
|-<br />
| Less duplication of metadata (c.f. PXML files) || || || ||<br />
|-<br />
| AUR-like system:<br />
* Building recipes is easier to replicate and contribute<br />
* If maintainer is jerk, moderator can move his maintanership<br />
|| Fork makepkg || || [http://boards.openpandora.org/topic/15530-the-software-side/page-7#entry310691 Cloudef is working on this task] ||<br />
|-<br />
| A compatibility layer for the current *.PND system:<br />
* The more transparent, the better<br />
||<br />
* Dual-boot<br />
* Virtual Machine<br />
* Tune Debian system to include all libraries used in current SuperZaxxon<br />
** OpenEmbedded it's based on the Debian, afterall<br />
** Probably won't happen because of hard/soft float point library issues<br />
* chroot<br />
** chroot doesn't start services like the real system, then the services of the chroot must be replaced/configured<br />
|| High || ||<br />
* Pyra will have larger internal space, then duplicate system isn't a large problem<br />
* The compatibility layer should have its own table?<br />
|-<br />
| Cross-building? || || || Undecided ||<br />
|-<br />
| Home-dir of applications ||<br />
* Maybe one per SD-card<br />
* Union-fs? Aufs?<br />
* overridable by env var?<br />
|| || ||<br />
|-<br />
| Overridable scripts ||<br />
* By creating files with the right name, you can override files in the package (e.g. scripts)<br />
* Yes:<br />
** Cool for hackers<br />
* No:<br />
** Necessitates a union filesystem<br />
** Not absolutely necessary - hackers can just rebuild the package.<br />
|| || ||<br />
|-<br />
| File-extension? ||<br />
* .pyra<br />
** We're not using fucking DOS for lord-sake, what's the point of use 3-letters extension?<br />
* .pyd for pyra-data<br />
* .pyr<br />
* .pp for pyra program<br />
* .pa for pyra app<br />
* .drp for dragonbox pyra<br />
** Why limit this to just the Pyra? Could be useful in future systems/elsewhere too.<br />
* ...<br />
|| || undecided. || Maybe a series of polls on the boards? ;)<br />
|-<br />
| Keep what is good || Document all of the good features and port the content to this table || High || Not started ||<br />
|-<br />
| Extra bonus points for a design that can easily be used on other distros and devices as well || || Optional || Won't fix ||<br />
* .deb dependencies already kills the system-agnostic system<br />
** Unless we submit our changes to upstream Debian, which is unlikely to happen<br />
|-<br />
| Multiple binaries/icons per app || || || || Somebody (lost on the sea of comments) didn't like this design, but it should be possible for special software<br />
|-<br />
| Delta packages || || || || http://boards.openpandora.org/topic/15530-the-software-side/page-9#entry311600<br />
|-<br />
| PATH handling || || || || http://boards.openpandora.org/topic/15530-the-software-side/page-9#entry311618<br />
|}<br />
<br />
[[Category:PND]]<br />
[[Category:Ideas and proposals]]</div>Vinipsmakerhttps://pandorawiki.org/index.php?title=New_PND_format&diff=29481New PND format2014-02-06T22:45:24Z<p>Vinipsmaker: removing forgotten trash</p>
<hr />
<div>The current PND format has some shortcomings as listed below. This page should serve as a discussion page/white board for how the format could be improved.<br />
<br />
''' Proposal for discussion, this is just some opening shots '''<br />
<br />
== Current situation ==<br />
The current (ISO-based) PND format has the following shortcomings:<br />
<br />
* It currently uses ISO, SquashFS, and other filesystems... This means that there's no standard to follow, and libpnd needs to carry around support for a multitude of file systems.<br />
* It (often, but not always) uses the [http://en.wikipedia.org/wiki/ISO_9660 ISO file system] which is inefficient at storing data, because:<br />
** It contains duplicate headers for each file entry[http://users.telenet.be/it3.consultants.bvba/handouts/ISO9960.html] (one version with big-endian integers, one version with little-endian integers[http://en.wikipedia.org/wiki/Endianness]) which leads to many kilobytes of wasted space.<br />
** Its file tables are fixed-size, and there are therefore limitations on how many folders you can have, what names you can give your files, how big your files can be etc. For instance, files can only have names that are max 31 characters long, all upper-case, and limited to the ASCII character encoding, and an ISO does only support a folder depth of 8[http://www.on-time.com/rtos-32-docs/rtfiles-32/programming-manual/iso-9660/]. To support more, the Joliet file system extension is needed (or isofs won't recognize all file paths), and the Joliet header can be placed practically anywhere in the file, which means additional seek times.<br />
** It needs various file system extensions to behave correctly, like for example [http://en.wikipedia.org/wiki/Rock_Ridge Rock Ridge Interchange Protocol] and [http://en.wikipedia.org/wiki/Joliet_(file_system) Joliet]. Without them, it becomes pretty useless as demonstrated above, and with them, it becomes difficult to read by a tool (You can only use isofs! There are no other tools out there for programmers to use!).<br />
** It's very difficult to use since you need special ISO making tools to create the image. ISO is a relatively rare format if you're not technically inclined.<br />
** If you want to add or remove files from it, it's near impossible if you use a compact ISO file, since there's no way that you can "expand" an existing ISO file.<br />
* The PND header data is at the end (or in the middle of the file if a screenshot is included) which makes it impossible for e.g. libmagic to recognize the PND file. It will instead recognize it as an ISO file. Having the header data at the end also makes it take a very long time to find the data, making tools and the libpnd library very inefficient.<br />
* The PND file uses a custom XML format for its metadata. There's no reason to do this, especially since the established ".desktop" file format fills exactly the same function.<br />
(note: ".desktop" does not contain all metadata we have wanted over time, but it's of course possible to add extensions à la "X-Pandora-Whatever=xyz")<br />
* There's no "index" for the PND file. The whole file has to be scanned (albeit backwards) to find a PXML file, and there's a big chance for false positives etc.<br />
* Data is just appended linearly to the file so there's no order. If the format is to be extended (to e.g. include an icon file after the screenshot file), should the data just be appended as well?<br />
* UTF-8 is strictly the only encoding that is supported. If you make your PXML on a Windows machine, it won't work.<br />
<br />
== Proposed revisions ==<br />
=== Step 1: File system ===<br />
The file system for PNDs should be replaced by the ''uncompressed'' ZIP archive format. ZIP has the advantage that it's incredibly compact, and uncompressed ZIP makes it possible to read data from the file without having to do any decompression.<br />
<br />
ZIP files are mountable using various implementations of zipfs, and most of these implementations won't store files in memory when they are being read if the ZIP file is uncompressed.<br />
<br />
=== Step 2: Metadata ===<br />
PND files should no longer require special tools for them to be created. Therefore, since ZIP files support random access on files, it shouldn't be necessary to append/prepend metadata to the file. The user simply includes the "PXML.xml" and "preview.png" files inside of the ZIP, and any tools that need information about the PND can simply go to the central directory of the ZIP file (or use a simple ZIP library to do it for them) and get the location of the file inside of the ZIP. This should also dramatically decrease loading times for PNDs.<br />
<br />
Note; performance testing is needed; at the time we started, zip was enormously slower than plain ISO; with driver changes, it may or may not be so, hence our adopted multiple-filesystem-type system. zipfs is one possible option.<br />
<br />
=== Step 3: Metadata format ===<br />
Current PND files include so-called "PXML.xml" files. These files have a custom XML format that has a strange structure.<br />
<br />
These files should be replaced by ".desktop" files. A PND can contain one or more ".desktop" files in its root directory that specify how an application should be launched. PND tools simply use all ".desktop" files they can find in the PND when creating launchers for the contents of the PND.<br />
<br />
== Advantages ==<br />
* There don't have to be ''any'' special tools for reading PND files. The package can be run on any platform using any programming language that can read ZIP files.<br />
* We can use existing facilities to manage the launching of applications. The ".desktop" files can basically be copied without modification into standard locations of the system, and all launchers will become aware of them.<br />
* Reading PND files becomes easier (because of better tool support... sorry, but you can't link to libpnd in all programming languages) and quicker (becuase of the central directory for random file access).<br />
<br />
=== Benchmarks ===<br />
A benchmark was carried through to measure the performance of various package implementations.<br />
The compared systems were:<br />
* ZIP-based uncompressed packages using fuse-zip (a zipfs implementation) to read the files<br />
* ZIP-based compressed packages also using fuse-zip.<br />
* ISO8859-based packages using isofs<br />
* CramFS-based packages.<br />
<br />
A total of 9 files were generated for the test. They have sizes 1, 2, 3, 4, 16, 32, 64, 128 and 256 MB, and were generated via "dd if=/dev/urandom of=$file bs=1M count=$size".<br />
<br />
Because the files contain random data, they did not compress very well. The resulting files were:<br />
* "zipcompressed.zip" 511.1 MB<br />
* "zipuncompressed.zip" 511.0 MB<br />
* "iso.iso" 511.3 MB<br />
* "cramfs.image" 95.4 MB (WOW!)<br />
<br />
Some notes:<br />
* No write tests were done for obvious reasons<br />
* No random access tests were done since the Pandora will use SD cards and thus not get penalized from random access, and that's not what we're interested in anyways.<br />
* Before each test, the commands "sync; echo 3 > /proc/sys/vm/drop_caches" were run.<br />
<br />
==== First test: linear read time ====<br />
Command: "time cat mountpoint/* > /dev/null"<br />
<br />
* ISO: 8.119 seconds<br />
* CramFS: 1.488 seconds (WOW!)<br />
* Zip (compressed): 8.535 seconds<br />
* Zip (uncompressed): 8.290 seconds<br />
<br />
==== Second test: RAM usage ====<br />
-- TODO, have to find a good method of measuring this --<br />
<br />
== Remaining issues ==<br />
* How should libpnd tell the difference between old and new PND files? Should it depend on the "file" tool, should it run its own recognition algorithm, or what? Should the extension be changed to e.g. ".box" (Which was proposed in a thread)?<br />
* How to make sure that the ZIP files are uncompressed? Should we provide a script that "uncompresses" an ordinary ZIP file?<br />
* Is cramfs better than ZIP?<br />
** cramfs cannot be uncompressed like ZIP can<br />
** If you compare compressed ZIP and cramfs, cramfs is more efficient.<br />
** cramfs is harder to use than ZIP (for the developer).<br />
** cramfs ''needs'' to have all of its opened files in memory.<br />
<br />
== Upgrade path from the old PND format ==<br />
A simple script can be written that extracts the old PND file, including the screenshot and the PXML.xml file. The PXML.xml file is then converted to one or many .desktop files, and the desktop files, the preview.png file, and the package contents are compacted into a ZIP that then is renamed to "*.pnd"<br />
<br />
== Usage scenario ==<br />
=== Repackaging of an application from another package format ===<br />
* The user grabs the package for the application<br />
* He dumps the executable and all required libraries into a folder<br />
* He dumps a "screenshot.png" file into the folder that he's made<br />
* He copies the ".desktop" file(s) for the application from the old package, opens them, replaces "Exec=/usr/bin/bla" with "Exec=./bla" and saves them in the directory<br />
* He uses a ZIP archiver to make a ZIP out of the folder, making sure that he sets the "uncompressed" option.<br />
<br />
=== Creating PNDs as part of a build process ===<br />
* The build tool creates a directory with all of the necessary information like above and invokes the "zip" utility to compress the folder.<br />
<br />
=== Accessing a PND's contents for the... ===<br />
<br />
==== ...lazy programmer ====<br />
* The programmer extracts the PND into a folder via libzip and accesses its contents.<br />
<br />
==== ...pragmatic programmer ====<br />
* The programmer uses zipfs to mount the ZIP and accesses the file's contents.<br />
<br />
==== ...smart performance-aware programmer ====<br />
* The programmer fseek()'s the ZIP file until he finds 0x04034b50.<br />
* He jumps 18 bytes forward and reads the int at that location, storing it in "length".<br />
* He jumps 4 bytes and checks if this int matches "length". If it doesn't, it means that the file is compressed, and an error is reported.<br />
* He then jumps forward 4 bytes and stores the short at that location in a variable "nameLength".<br />
* Another 2-byte jump gets the short "extensionsLength".<br />
* He then jumps 2 bytes and reads "nameLength" amount of bytes from the file.<br />
* He then uses strcmp() to see if this string matches the sought-after file. If not, he continues the fseek().<br />
* The programmer now skips "entensionsLength" amount of bytes.<br />
* The programmer reads "length" amount of bytes from the file, and uses this as the sought-after file data.<br />
<br />
==== ...performance fascist/programmer who wants low seek times ====<br />
* The programmer fseek()'s the file from the end until he finds 0x02014b50.<br />
* He then uses the following table to get the information he needs:<br />
{|class="wikitable"<br />
|+ ZIP central directory file header<br />
|-<br />
! Offset !! Bytes !! Description<br />
|-<br />
| 0 || 4 || Central directory file header signature = 0x02014b50<br />
|-<br />
| 4 || 2 || Version made by<br />
|-<br />
| 6 || 2 || Version needed to extract (minimum)<br />
|-<br />
| 8 || 2 || General purpose bit flag<br />
|-<br />
| 10 || 2 || Compression method<br />
|-<br />
| 12 || 2 || File last modification time<br />
|-<br />
| 14 || 2 || File last modification date<br />
|-<br />
| 16 || 4 || CRC-32<br />
|-<br />
| 20 || 4 || Compressed size<br />
|-<br />
| 24 || 4 || Uncompressed size<br />
|-<br />
| 28 || 2 || File name length (''n'')<br />
|-<br />
| 30 || 2 || Extra field length (''m'')<br />
|-<br />
| 32 || 2 || File comment length (''k'')<br />
|-<br />
| 34 || 2 || Disk number where file starts<br />
|-<br />
| 36 || 2 || Internal file attributes<br />
|-<br />
| 38 || 4 || External file attributes<br />
|-<br />
| 42 || 4 || Relative offset of local file header<br />
|-<br />
| 46 || ''n'' || File name<br />
|-<br />
| 46+''n'' || ''m'' || Extra field<br />
|-<br />
| 46+''n''+''m'' || ''k'' || File comment<br />
|}<br />
<br />
* The relative file offset can then be used to jump to the file in question. The file is then scanned as described in the previous section.<br />
<br />
== The 2014/Pyra effort ==<br />
<br />
A new format is being developed for the DragonBox Pyra handheld. Here are the requirements/notes:<br />
<br />
The compatibility layer should have its own table?<br />
<br />
{| class="wikitable"<br />
|-<br />
! Requirement !! Solutions !! Importance !! Status (decided, undecided, post-proned, won't fix, ...) !! Notes<br />
|-<br />
| Should the new format be supported by Pandora AND Pyra? ||<br />
* Yes<br />
** Makes releasing compatible applications easier.<br />
* no<br />
** Pandora is softfp and Pyra is hardfp - it's unlikely one package could work on both systems anyway.<br />
|| Very high || ||<br />
* http://boards.openpandora.org/topic/15530-the-software-side/page-7#entry310691<br />
* This decision affects several of the requirements and possible solutions on this table and should be decided sooner<br />
|-<br />
| API/Daemon to switch between joystick/keyboard mode || || || ||<br />
* Not exactly a PND-error<br />
* aTc gathered a nice list of notes: http://boards.openpandora.org/topic/15587-userspace-input-daemon/#entry311623<br />
|-<br />
| Guideline for consistent use of gaming buttons:<br />
* It should reduce the need of patches on top of the ported software<br />
||<br />
* A keyboard/joystick remapping function<br />
* A guideline for software developers<br />
|| || || No single-solution, but a set of them<br />
|-<br />
| Easier to parse than current PNDs and less fragile ||<br />
* header + metadata (UBJSON) + icon + screenshots + squashfs<br />
* tar archive<br />
** more future proof<br />
** easier for shell scripts<br />
*** with dd you can extract the tar part and so<br />
** filenames and file permissions are overhead<br />
*** But the overhead is too small to be significant?<br />
** easier to modify after-compiled<br />
*** not really a problem, because the user should be contributing to the recipe anyway<br />
*** a specialized tool can be create if another solution is chosen<br />
* zipfs uncompressed filesystem<br />
** See above sections<br />
* ar archive<br />
** Like .deb format<br />
|| || ||<br />
|-<br />
| Should the metadata include packages dependencies? ||<br />
* Yes<br />
** apt-get install package --print-uris will print download uris for dep packages. Using this uris download packages and extract all (dpkg -x 1.deb ./extracted)<br />
*** Not enough. It'll need two passes between an online and an offline system. It's impossible to be sure that an offline installation will work in one pass without including the whole repo: http://boards.openpandora.org/topic/15530-the-software-side/page-9#entry311602<br />
** EvilDragon voted for yes-and-no? http://boards.openpandora.org/topic/15530-the-software-side/page-5#entry310590<br />
*** Or does it mean a minimum "os-version required" metadata on the package and let the user update the system to have the latest libraries available?<br />
*** Possibly he doesn't care and just want something that just works: http://boards.openpandora.org/topic/15574-confliting-requirements/page-2#entry311580<br />
** http://boards.openpandora.org/topic/15530-the-software-side/page-5#entry310627<br />
* No.<br />
** EvilDragon commented that he wanna a plug-and-play-and-works system ( http://boards.openpandora.org/topic/15530-the-software-side/page-3#entry310412 )<br />
** If the user is lacking of internet access, "plug-and-play" package won't work<br />
** .deb for general-purpose software<br />
* Complete user-space like containment features (see http://www.docker.io/ for an example)<br />
|| || Decided ||<br />
* After a long heated discussion, general consensus seems to be:<br />
** Tune the makepnd script to create flat PNDs (packages which only deps are the minimum required set already present) and light PNDs (packages with .deb dependencies).<br />
*** Both packages would be included in the repo and user download what he likes more.<br />
|-<br />
| Minimum set of dependencies (SDL, ...) || || || WIP || Progress is happenning on github:<br />
* https://github.com/Pyra-Handheld/TODO/issues/5<br />
* https://github.com/Pyra-Handheld/TODO/issues/7<br />
|-<br />
| It should be click-and-run ||<br />
* binfmt<br />
* Sheebang<br />
* File (extension) association<br />
|| || ||<br />
|-<br />
| Less duplication of metadata (c.f. PXML files) || || || ||<br />
|-<br />
| AUR-like system:<br />
* Building recipes is easier to replicate and contribute<br />
* If maintainer is jerk, moderator can move his maintanership<br />
|| Fork makepkg || || [http://boards.openpandora.org/topic/15530-the-software-side/page-7#entry310691 Cloudef is working on this task] ||<br />
|-<br />
| A compatibility layer for the current *.PND system:<br />
* The more transparent, the better<br />
||<br />
* Dual-boot<br />
* Virtual Machine<br />
* Tune Debian system to include all libraries used in current SuperZaxxon<br />
** OpenEmbedded it's based on the Debian, afterall<br />
** Probably won't happen because of hard/soft float point library issues<br />
* chroot<br />
** chroot doesn't start services like the real system, then the services of the chroot must be replaced/configured<br />
|| High || ||<br />
* Pyra will have larger internal space, then duplicate system isn't a large problem<br />
* For current OpenPandora, we can already run Debian on SD-card.<br />
** Maybe we can provide Pyra apps for OpenPandora using the Debian-on-sd-card opposed to Debian-on-internal-space<br />
|-<br />
| Cross-building? || || || Undecided ||<br />
|-<br />
| Home-dir of applications ||<br />
* Maybe one per SD-card<br />
* Union-fs? Aufs?<br />
* overridable by env var?<br />
|| || ||<br />
|-<br />
| Overridable scripts ||<br />
* By creating files with the right name, you can override files in the package (e.g. scripts)<br />
* Yes:<br />
** Cool for hackers<br />
* No:<br />
** Necessitates a union filesystem<br />
** Not absolutely necessary - hackers can just rebuild the package.<br />
|| || ||<br />
|-<br />
| File-extension? ||<br />
* .pyra<br />
** We're not using fucking DOS for lord-sake, what's the point of use 3-letters extension?<br />
* .pyd for pyra-data<br />
* .pyr<br />
* .pp for pyra program<br />
* .pa for pyra app<br />
* .drp for dragonbox pyra<br />
** Why limit this to just the Pyra? Could be useful in future systems/elsewhere too.<br />
* ...<br />
|| || undecided. || Maybe a series of polls on the boards? ;)<br />
|-<br />
| Keep what is good || Document all of the good features and port the content to this table || High || Not started ||<br />
|-<br />
| Extra bonus points for a design that can easily be used on other distros and devices as well || || Optional || Won't fix ||<br />
* .deb dependencies already kills the system-agnostic system<br />
** Unless we submit our changes to upstream Debian, which is unlikely to happen<br />
|-<br />
| Multiple binaries/icons per app || || || || Somebody (lost on the sea of comments) didn't like this design, but it should be possible for special software<br />
|-<br />
| Delta packages || || || || http://boards.openpandora.org/topic/15530-the-software-side/page-9#entry311600<br />
|-<br />
| PATH handling || || || || http://boards.openpandora.org/topic/15530-the-software-side/page-9#entry311618<br />
|}<br />
<br />
[[Category:PND]]<br />
[[Category:Ideas and proposals]]</div>Vinipsmakerhttps://pandorawiki.org/index.php?title=New_PND_format&diff=29480New PND format2014-02-06T22:44:05Z<p>Vinipsmaker: mark deps as decided</p>
<hr />
<div>The current PND format has some shortcomings as listed below. This page should serve as a discussion page/white board for how the format could be improved.<br />
<br />
''' Proposal for discussion, this is just some opening shots '''<br />
<br />
== Current situation ==<br />
The current (ISO-based) PND format has the following shortcomings:<br />
<br />
* It currently uses ISO, SquashFS, and other filesystems... This means that there's no standard to follow, and libpnd needs to carry around support for a multitude of file systems.<br />
* It (often, but not always) uses the [http://en.wikipedia.org/wiki/ISO_9660 ISO file system] which is inefficient at storing data, because:<br />
** It contains duplicate headers for each file entry[http://users.telenet.be/it3.consultants.bvba/handouts/ISO9960.html] (one version with big-endian integers, one version with little-endian integers[http://en.wikipedia.org/wiki/Endianness]) which leads to many kilobytes of wasted space.<br />
** Its file tables are fixed-size, and there are therefore limitations on how many folders you can have, what names you can give your files, how big your files can be etc. For instance, files can only have names that are max 31 characters long, all upper-case, and limited to the ASCII character encoding, and an ISO does only support a folder depth of 8[http://www.on-time.com/rtos-32-docs/rtfiles-32/programming-manual/iso-9660/]. To support more, the Joliet file system extension is needed (or isofs won't recognize all file paths), and the Joliet header can be placed practically anywhere in the file, which means additional seek times.<br />
** It needs various file system extensions to behave correctly, like for example [http://en.wikipedia.org/wiki/Rock_Ridge Rock Ridge Interchange Protocol] and [http://en.wikipedia.org/wiki/Joliet_(file_system) Joliet]. Without them, it becomes pretty useless as demonstrated above, and with them, it becomes difficult to read by a tool (You can only use isofs! There are no other tools out there for programmers to use!).<br />
** It's very difficult to use since you need special ISO making tools to create the image. ISO is a relatively rare format if you're not technically inclined.<br />
** If you want to add or remove files from it, it's near impossible if you use a compact ISO file, since there's no way that you can "expand" an existing ISO file.<br />
* The PND header data is at the end (or in the middle of the file if a screenshot is included) which makes it impossible for e.g. libmagic to recognize the PND file. It will instead recognize it as an ISO file. Having the header data at the end also makes it take a very long time to find the data, making tools and the libpnd library very inefficient.<br />
* The PND file uses a custom XML format for its metadata. There's no reason to do this, especially since the established ".desktop" file format fills exactly the same function.<br />
(note: ".desktop" does not contain all metadata we have wanted over time, but it's of course possible to add extensions à la "X-Pandora-Whatever=xyz")<br />
* There's no "index" for the PND file. The whole file has to be scanned (albeit backwards) to find a PXML file, and there's a big chance for false positives etc.<br />
* Data is just appended linearly to the file so there's no order. If the format is to be extended (to e.g. include an icon file after the screenshot file), should the data just be appended as well?<br />
* UTF-8 is strictly the only encoding that is supported. If you make your PXML on a Windows machine, it won't work.<br />
<br />
== Proposed revisions ==<br />
=== Step 1: File system ===<br />
The file system for PNDs should be replaced by the ''uncompressed'' ZIP archive format. ZIP has the advantage that it's incredibly compact, and uncompressed ZIP makes it possible to read data from the file without having to do any decompression.<br />
<br />
ZIP files are mountable using various implementations of zipfs, and most of these implementations won't store files in memory when they are being read if the ZIP file is uncompressed.<br />
<br />
=== Step 2: Metadata ===<br />
PND files should no longer require special tools for them to be created. Therefore, since ZIP files support random access on files, it shouldn't be necessary to append/prepend metadata to the file. The user simply includes the "PXML.xml" and "preview.png" files inside of the ZIP, and any tools that need information about the PND can simply go to the central directory of the ZIP file (or use a simple ZIP library to do it for them) and get the location of the file inside of the ZIP. This should also dramatically decrease loading times for PNDs.<br />
<br />
Note; performance testing is needed; at the time we started, zip was enormously slower than plain ISO; with driver changes, it may or may not be so, hence our adopted multiple-filesystem-type system. zipfs is one possible option.<br />
<br />
=== Step 3: Metadata format ===<br />
Current PND files include so-called "PXML.xml" files. These files have a custom XML format that has a strange structure.<br />
<br />
These files should be replaced by ".desktop" files. A PND can contain one or more ".desktop" files in its root directory that specify how an application should be launched. PND tools simply use all ".desktop" files they can find in the PND when creating launchers for the contents of the PND.<br />
<br />
== Advantages ==<br />
* There don't have to be ''any'' special tools for reading PND files. The package can be run on any platform using any programming language that can read ZIP files.<br />
* We can use existing facilities to manage the launching of applications. The ".desktop" files can basically be copied without modification into standard locations of the system, and all launchers will become aware of them.<br />
* Reading PND files becomes easier (because of better tool support... sorry, but you can't link to libpnd in all programming languages) and quicker (becuase of the central directory for random file access).<br />
<br />
=== Benchmarks ===<br />
A benchmark was carried through to measure the performance of various package implementations.<br />
The compared systems were:<br />
* ZIP-based uncompressed packages using fuse-zip (a zipfs implementation) to read the files<br />
* ZIP-based compressed packages also using fuse-zip.<br />
* ISO8859-based packages using isofs<br />
* CramFS-based packages.<br />
<br />
A total of 9 files were generated for the test. They have sizes 1, 2, 3, 4, 16, 32, 64, 128 and 256 MB, and were generated via "dd if=/dev/urandom of=$file bs=1M count=$size".<br />
<br />
Because the files contain random data, they did not compress very well. The resulting files were:<br />
* "zipcompressed.zip" 511.1 MB<br />
* "zipuncompressed.zip" 511.0 MB<br />
* "iso.iso" 511.3 MB<br />
* "cramfs.image" 95.4 MB (WOW!)<br />
<br />
Some notes:<br />
* No write tests were done for obvious reasons<br />
* No random access tests were done since the Pandora will use SD cards and thus not get penalized from random access, and that's not what we're interested in anyways.<br />
* Before each test, the commands "sync; echo 3 > /proc/sys/vm/drop_caches" were run.<br />
<br />
==== First test: linear read time ====<br />
Command: "time cat mountpoint/* > /dev/null"<br />
<br />
* ISO: 8.119 seconds<br />
* CramFS: 1.488 seconds (WOW!)<br />
* Zip (compressed): 8.535 seconds<br />
* Zip (uncompressed): 8.290 seconds<br />
<br />
==== Second test: RAM usage ====<br />
-- TODO, have to find a good method of measuring this --<br />
<br />
== Remaining issues ==<br />
* How should libpnd tell the difference between old and new PND files? Should it depend on the "file" tool, should it run its own recognition algorithm, or what? Should the extension be changed to e.g. ".box" (Which was proposed in a thread)?<br />
* How to make sure that the ZIP files are uncompressed? Should we provide a script that "uncompresses" an ordinary ZIP file?<br />
* Is cramfs better than ZIP?<br />
** cramfs cannot be uncompressed like ZIP can<br />
** If you compare compressed ZIP and cramfs, cramfs is more efficient.<br />
** cramfs is harder to use than ZIP (for the developer).<br />
** cramfs ''needs'' to have all of its opened files in memory.<br />
<br />
== Upgrade path from the old PND format ==<br />
A simple script can be written that extracts the old PND file, including the screenshot and the PXML.xml file. The PXML.xml file is then converted to one or many .desktop files, and the desktop files, the preview.png file, and the package contents are compacted into a ZIP that then is renamed to "*.pnd"<br />
<br />
== Usage scenario ==<br />
=== Repackaging of an application from another package format ===<br />
* The user grabs the package for the application<br />
* He dumps the executable and all required libraries into a folder<br />
* He dumps a "screenshot.png" file into the folder that he's made<br />
* He copies the ".desktop" file(s) for the application from the old package, opens them, replaces "Exec=/usr/bin/bla" with "Exec=./bla" and saves them in the directory<br />
* He uses a ZIP archiver to make a ZIP out of the folder, making sure that he sets the "uncompressed" option.<br />
<br />
=== Creating PNDs as part of a build process ===<br />
* The build tool creates a directory with all of the necessary information like above and invokes the "zip" utility to compress the folder.<br />
<br />
=== Accessing a PND's contents for the... ===<br />
<br />
==== ...lazy programmer ====<br />
* The programmer extracts the PND into a folder via libzip and accesses its contents.<br />
<br />
==== ...pragmatic programmer ====<br />
* The programmer uses zipfs to mount the ZIP and accesses the file's contents.<br />
<br />
==== ...smart performance-aware programmer ====<br />
* The programmer fseek()'s the ZIP file until he finds 0x04034b50.<br />
* He jumps 18 bytes forward and reads the int at that location, storing it in "length".<br />
* He jumps 4 bytes and checks if this int matches "length". If it doesn't, it means that the file is compressed, and an error is reported.<br />
* He then jumps forward 4 bytes and stores the short at that location in a variable "nameLength".<br />
* Another 2-byte jump gets the short "extensionsLength".<br />
* He then jumps 2 bytes and reads "nameLength" amount of bytes from the file.<br />
* He then uses strcmp() to see if this string matches the sought-after file. If not, he continues the fseek().<br />
* The programmer now skips "entensionsLength" amount of bytes.<br />
* The programmer reads "length" amount of bytes from the file, and uses this as the sought-after file data.<br />
<br />
==== ...performance fascist/programmer who wants low seek times ====<br />
* The programmer fseek()'s the file from the end until he finds 0x02014b50.<br />
* He then uses the following table to get the information he needs:<br />
{|class="wikitable"<br />
|+ ZIP central directory file header<br />
|-<br />
! Offset !! Bytes !! Description<br />
|-<br />
| 0 || 4 || Central directory file header signature = 0x02014b50<br />
|-<br />
| 4 || 2 || Version made by<br />
|-<br />
| 6 || 2 || Version needed to extract (minimum)<br />
|-<br />
| 8 || 2 || General purpose bit flag<br />
|-<br />
| 10 || 2 || Compression method<br />
|-<br />
| 12 || 2 || File last modification time<br />
|-<br />
| 14 || 2 || File last modification date<br />
|-<br />
| 16 || 4 || CRC-32<br />
|-<br />
| 20 || 4 || Compressed size<br />
|-<br />
| 24 || 4 || Uncompressed size<br />
|-<br />
| 28 || 2 || File name length (''n'')<br />
|-<br />
| 30 || 2 || Extra field length (''m'')<br />
|-<br />
| 32 || 2 || File comment length (''k'')<br />
|-<br />
| 34 || 2 || Disk number where file starts<br />
|-<br />
| 36 || 2 || Internal file attributes<br />
|-<br />
| 38 || 4 || External file attributes<br />
|-<br />
| 42 || 4 || Relative offset of local file header<br />
|-<br />
| 46 || ''n'' || File name<br />
|-<br />
| 46+''n'' || ''m'' || Extra field<br />
|-<br />
| 46+''n''+''m'' || ''k'' || File comment<br />
|}<br />
<br />
* The relative file offset can then be used to jump to the file in question. The file is then scanned as described in the previous section.<br />
<br />
== The 2014/Pyra effort ==<br />
<br />
A new format is being developed for the DragonBox Pyra handheld. Here are the requirements/notes:<br />
<br />
The compatibility layer should have its own table?<br />
<br />
{| class="wikitable"<br />
|-<br />
! Requirement !! Solutions !! Importance !! Status (decided, undecided, post-proned, won't fix, ...) !! Notes<br />
|-<br />
| Should the new format be supported by Pandora AND Pyra? ||<br />
* Yes<br />
** Makes releasing compatible applications easier.<br />
* no<br />
** Pandora is softfp and Pyra is hardfp - it's unlikely one package could work on both systems anyway.<br />
|| Very high || ||<br />
* http://boards.openpandora.org/topic/15530-the-software-side/page-7#entry310691<br />
* This decision affects several of the requirements and possible solutions on this table and should be decided sooner<br />
|-<br />
| API/Daemon to switch between joystick/keyboard mode || || || ||<br />
* Not exactly a PND-error<br />
* aTc gathered a nice list of notes: http://boards.openpandora.org/topic/15587-userspace-input-daemon/#entry311623<br />
|-<br />
| Guideline for consistent use of gaming buttons:<br />
* It should reduce the need of patches on top of the ported software<br />
||<br />
* A keyboard/joystick remapping function<br />
* A guideline for software developers<br />
|| || || No single-solution, but a set of them<br />
|-<br />
| Easier to parse than current PNDs and less fragile ||<br />
* header + metadata (UBJSON) + icon + screenshots + squashfs<br />
* tar archive<br />
** more future proof<br />
** easier for shell scripts<br />
*** with dd you can extract the tar part and so<br />
** filenames and file permissions are overhead<br />
*** But the overhead is too small to be significant?<br />
** easier to modify after-compiled<br />
*** not really a problem, because the user should be contributing to the recipe anyway<br />
*** a specialized tool can be create if another solution is chosen<br />
* zipfs uncompressed filesystem<br />
** See above sections<br />
* ar archive<br />
** Like .deb format<br />
|| || ||<br />
|-<br />
| Should the metadata include packages dependencies? ||<br />
* Yes<br />
** apt-get install package --print-uris will print download uris for dep packages. Using this uris download packages and extract all (dpkg -x 1.deb ./extracted)<br />
*** Not enough. It'll need two passes between an online and an offline system. It's impossible to be sure that an offline installation will work in one pass without including the whole repo: http://boards.openpandora.org/topic/15530-the-software-side/page-9#entry311602<br />
** EvilDragon voted for yes-and-no? http://boards.openpandora.org/topic/15530-the-software-side/page-5#entry310590<br />
*** Or does it mean a minimum "os-version required" metadata on the package and let the user update the system to have the latest libraries available?<br />
*** Possibly he doesn't care and just want something that just works: http://boards.openpandora.org/topic/15574-confliting-requirements/page-2#entry311580<br />
** http://boards.openpandora.org/topic/15530-the-software-side/page-5#entry310627<br />
* No.<br />
** EvilDragon commented that he wanna a plug-and-play-and-works system ( http://boards.openpandora.org/topic/15530-the-software-side/page-3#entry310412 )<br />
** If the user is lacking of internet access, "plug-and-play" package won't work<br />
** .deb for general-purpose software<br />
* Complete user-space like containment features (see http://www.docker.io/ for an example)<br />
|| || Decided ||<br />
* After a long heated discussion, general consensus seems to be:<br />
** Tune the makepnd script to create flat PNDs (packages which only deps are the minimum required set already present) and light PNDs (packages with .deb dependencies).<br />
*** Both packages would be included in the repo and user download what he likes more.<br />
* Independent of what choice is made, the minimum set of dependencies is a good idea to have around<br />
|-<br />
| Minimum set of dependencies (SDL, ...) || || || WIP || Progress is happenning on github:<br />
* https://github.com/Pyra-Handheld/TODO/issues/5<br />
* https://github.com/Pyra-Handheld/TODO/issues/7<br />
|-<br />
| It should be click-and-run ||<br />
* binfmt<br />
* Sheebang<br />
* File (extension) association<br />
|| || ||<br />
|-<br />
| Less duplication of metadata (c.f. PXML files) || || || ||<br />
|-<br />
| AUR-like system:<br />
* Building recipes is easier to replicate and contribute<br />
* If maintainer is jerk, moderator can move his maintanership<br />
|| Fork makepkg || || [http://boards.openpandora.org/topic/15530-the-software-side/page-7#entry310691 Cloudef is working on this task] ||<br />
|-<br />
| A compatibility layer for the current *.PND system:<br />
* The more transparent, the better<br />
||<br />
* Dual-boot<br />
* Virtual Machine<br />
* Tune Debian system to include all libraries used in current SuperZaxxon<br />
** OpenEmbedded it's based on the Debian, afterall<br />
** Probably won't happen because of hard/soft float point library issues<br />
* chroot<br />
** chroot doesn't start services like the real system, then the services of the chroot must be replaced/configured<br />
|| High || ||<br />
* Pyra will have larger internal space, then duplicate system isn't a large problem<br />
* For current OpenPandora, we can already run Debian on SD-card.<br />
** Maybe we can provide Pyra apps for OpenPandora using the Debian-on-sd-card opposed to Debian-on-internal-space<br />
|-<br />
| Cross-building? || || || Undecided ||<br />
|-<br />
| Home-dir of applications ||<br />
* Maybe one per SD-card<br />
* Union-fs? Aufs?<br />
* overridable by env var?<br />
|| || ||<br />
|-<br />
| Overridable scripts ||<br />
* By creating files with the right name, you can override files in the package (e.g. scripts)<br />
* Yes:<br />
** Cool for hackers<br />
* No:<br />
** Necessitates a union filesystem<br />
** Not absolutely necessary - hackers can just rebuild the package.<br />
|| || ||<br />
|-<br />
| File-extension? ||<br />
* .pyra<br />
** We're not using fucking DOS for lord-sake, what's the point of use 3-letters extension?<br />
* .pyd for pyra-data<br />
* .pyr<br />
* .pp for pyra program<br />
* .pa for pyra app<br />
* .drp for dragonbox pyra<br />
** Why limit this to just the Pyra? Could be useful in future systems/elsewhere too.<br />
* ...<br />
|| || undecided. || Maybe a series of polls on the boards? ;)<br />
|-<br />
| Keep what is good || Document all of the good features and port the content to this table || High || Not started ||<br />
|-<br />
| Extra bonus points for a design that can easily be used on other distros and devices as well || || Optional || Won't fix ||<br />
* .deb dependencies already kills the system-agnostic system<br />
** Unless we submit our changes to upstream Debian, which is unlikely to happen<br />
|-<br />
| Multiple binaries/icons per app || || || || Somebody (lost on the sea of comments) didn't like this design, but it should be possible for special software<br />
|-<br />
| Delta packages || || || || http://boards.openpandora.org/topic/15530-the-software-side/page-9#entry311600<br />
|-<br />
| PATH handling || || || || http://boards.openpandora.org/topic/15530-the-software-side/page-9#entry311618<br />
|}<br />
<br />
[[Category:PND]]<br />
[[Category:Ideas and proposals]]</div>Vinipsmakerhttps://pandorawiki.org/index.php?title=New_PND_format&diff=29479New PND format2014-02-06T04:46:23Z<p>Vinipsmaker: increasing section about deps</p>
<hr />
<div>The current PND format has some shortcomings as listed below. This page should serve as a discussion page/white board for how the format could be improved.<br />
<br />
''' Proposal for discussion, this is just some opening shots '''<br />
<br />
== Current situation ==<br />
The current (ISO-based) PND format has the following shortcomings:<br />
<br />
* It currently uses ISO, SquashFS, and other filesystems... This means that there's no standard to follow, and libpnd needs to carry around support for a multitude of file systems.<br />
* It (often, but not always) uses the [http://en.wikipedia.org/wiki/ISO_9660 ISO file system] which is inefficient at storing data, because:<br />
** It contains duplicate headers for each file entry[http://users.telenet.be/it3.consultants.bvba/handouts/ISO9960.html] (one version with big-endian integers, one version with little-endian integers[http://en.wikipedia.org/wiki/Endianness]) which leads to many kilobytes of wasted space.<br />
** Its file tables are fixed-size, and there are therefore limitations on how many folders you can have, what names you can give your files, how big your files can be etc. For instance, files can only have names that are max 31 characters long, all upper-case, and limited to the ASCII character encoding, and an ISO does only support a folder depth of 8[http://www.on-time.com/rtos-32-docs/rtfiles-32/programming-manual/iso-9660/]. To support more, the Joliet file system extension is needed (or isofs won't recognize all file paths), and the Joliet header can be placed practically anywhere in the file, which means additional seek times.<br />
** It needs various file system extensions to behave correctly, like for example [http://en.wikipedia.org/wiki/Rock_Ridge Rock Ridge Interchange Protocol] and [http://en.wikipedia.org/wiki/Joliet_(file_system) Joliet]. Without them, it becomes pretty useless as demonstrated above, and with them, it becomes difficult to read by a tool (You can only use isofs! There are no other tools out there for programmers to use!).<br />
** It's very difficult to use since you need special ISO making tools to create the image. ISO is a relatively rare format if you're not technically inclined.<br />
** If you want to add or remove files from it, it's near impossible if you use a compact ISO file, since there's no way that you can "expand" an existing ISO file.<br />
* The PND header data is at the end (or in the middle of the file if a screenshot is included) which makes it impossible for e.g. libmagic to recognize the PND file. It will instead recognize it as an ISO file. Having the header data at the end also makes it take a very long time to find the data, making tools and the libpnd library very inefficient.<br />
* The PND file uses a custom XML format for its metadata. There's no reason to do this, especially since the established ".desktop" file format fills exactly the same function.<br />
(note: ".desktop" does not contain all metadata we have wanted over time, but it's of course possible to add extensions à la "X-Pandora-Whatever=xyz")<br />
* There's no "index" for the PND file. The whole file has to be scanned (albeit backwards) to find a PXML file, and there's a big chance for false positives etc.<br />
* Data is just appended linearly to the file so there's no order. If the format is to be extended (to e.g. include an icon file after the screenshot file), should the data just be appended as well?<br />
* UTF-8 is strictly the only encoding that is supported. If you make your PXML on a Windows machine, it won't work.<br />
<br />
== Proposed revisions ==<br />
=== Step 1: File system ===<br />
The file system for PNDs should be replaced by the ''uncompressed'' ZIP archive format. ZIP has the advantage that it's incredibly compact, and uncompressed ZIP makes it possible to read data from the file without having to do any decompression.<br />
<br />
ZIP files are mountable using various implementations of zipfs, and most of these implementations won't store files in memory when they are being read if the ZIP file is uncompressed.<br />
<br />
=== Step 2: Metadata ===<br />
PND files should no longer require special tools for them to be created. Therefore, since ZIP files support random access on files, it shouldn't be necessary to append/prepend metadata to the file. The user simply includes the "PXML.xml" and "preview.png" files inside of the ZIP, and any tools that need information about the PND can simply go to the central directory of the ZIP file (or use a simple ZIP library to do it for them) and get the location of the file inside of the ZIP. This should also dramatically decrease loading times for PNDs.<br />
<br />
Note; performance testing is needed; at the time we started, zip was enormously slower than plain ISO; with driver changes, it may or may not be so, hence our adopted multiple-filesystem-type system. zipfs is one possible option.<br />
<br />
=== Step 3: Metadata format ===<br />
Current PND files include so-called "PXML.xml" files. These files have a custom XML format that has a strange structure.<br />
<br />
These files should be replaced by ".desktop" files. A PND can contain one or more ".desktop" files in its root directory that specify how an application should be launched. PND tools simply use all ".desktop" files they can find in the PND when creating launchers for the contents of the PND.<br />
<br />
== Advantages ==<br />
* There don't have to be ''any'' special tools for reading PND files. The package can be run on any platform using any programming language that can read ZIP files.<br />
* We can use existing facilities to manage the launching of applications. The ".desktop" files can basically be copied without modification into standard locations of the system, and all launchers will become aware of them.<br />
* Reading PND files becomes easier (because of better tool support... sorry, but you can't link to libpnd in all programming languages) and quicker (becuase of the central directory for random file access).<br />
<br />
=== Benchmarks ===<br />
A benchmark was carried through to measure the performance of various package implementations.<br />
The compared systems were:<br />
* ZIP-based uncompressed packages using fuse-zip (a zipfs implementation) to read the files<br />
* ZIP-based compressed packages also using fuse-zip.<br />
* ISO8859-based packages using isofs<br />
* CramFS-based packages.<br />
<br />
A total of 9 files were generated for the test. They have sizes 1, 2, 3, 4, 16, 32, 64, 128 and 256 MB, and were generated via "dd if=/dev/urandom of=$file bs=1M count=$size".<br />
<br />
Because the files contain random data, they did not compress very well. The resulting files were:<br />
* "zipcompressed.zip" 511.1 MB<br />
* "zipuncompressed.zip" 511.0 MB<br />
* "iso.iso" 511.3 MB<br />
* "cramfs.image" 95.4 MB (WOW!)<br />
<br />
Some notes:<br />
* No write tests were done for obvious reasons<br />
* No random access tests were done since the Pandora will use SD cards and thus not get penalized from random access, and that's not what we're interested in anyways.<br />
* Before each test, the commands "sync; echo 3 > /proc/sys/vm/drop_caches" were run.<br />
<br />
==== First test: linear read time ====<br />
Command: "time cat mountpoint/* > /dev/null"<br />
<br />
* ISO: 8.119 seconds<br />
* CramFS: 1.488 seconds (WOW!)<br />
* Zip (compressed): 8.535 seconds<br />
* Zip (uncompressed): 8.290 seconds<br />
<br />
==== Second test: RAM usage ====<br />
-- TODO, have to find a good method of measuring this --<br />
<br />
== Remaining issues ==<br />
* How should libpnd tell the difference between old and new PND files? Should it depend on the "file" tool, should it run its own recognition algorithm, or what? Should the extension be changed to e.g. ".box" (Which was proposed in a thread)?<br />
* How to make sure that the ZIP files are uncompressed? Should we provide a script that "uncompresses" an ordinary ZIP file?<br />
* Is cramfs better than ZIP?<br />
** cramfs cannot be uncompressed like ZIP can<br />
** If you compare compressed ZIP and cramfs, cramfs is more efficient.<br />
** cramfs is harder to use than ZIP (for the developer).<br />
** cramfs ''needs'' to have all of its opened files in memory.<br />
<br />
== Upgrade path from the old PND format ==<br />
A simple script can be written that extracts the old PND file, including the screenshot and the PXML.xml file. The PXML.xml file is then converted to one or many .desktop files, and the desktop files, the preview.png file, and the package contents are compacted into a ZIP that then is renamed to "*.pnd"<br />
<br />
== Usage scenario ==<br />
=== Repackaging of an application from another package format ===<br />
* The user grabs the package for the application<br />
* He dumps the executable and all required libraries into a folder<br />
* He dumps a "screenshot.png" file into the folder that he's made<br />
* He copies the ".desktop" file(s) for the application from the old package, opens them, replaces "Exec=/usr/bin/bla" with "Exec=./bla" and saves them in the directory<br />
* He uses a ZIP archiver to make a ZIP out of the folder, making sure that he sets the "uncompressed" option.<br />
<br />
=== Creating PNDs as part of a build process ===<br />
* The build tool creates a directory with all of the necessary information like above and invokes the "zip" utility to compress the folder.<br />
<br />
=== Accessing a PND's contents for the... ===<br />
<br />
==== ...lazy programmer ====<br />
* The programmer extracts the PND into a folder via libzip and accesses its contents.<br />
<br />
==== ...pragmatic programmer ====<br />
* The programmer uses zipfs to mount the ZIP and accesses the file's contents.<br />
<br />
==== ...smart performance-aware programmer ====<br />
* The programmer fseek()'s the ZIP file until he finds 0x04034b50.<br />
* He jumps 18 bytes forward and reads the int at that location, storing it in "length".<br />
* He jumps 4 bytes and checks if this int matches "length". If it doesn't, it means that the file is compressed, and an error is reported.<br />
* He then jumps forward 4 bytes and stores the short at that location in a variable "nameLength".<br />
* Another 2-byte jump gets the short "extensionsLength".<br />
* He then jumps 2 bytes and reads "nameLength" amount of bytes from the file.<br />
* He then uses strcmp() to see if this string matches the sought-after file. If not, he continues the fseek().<br />
* The programmer now skips "entensionsLength" amount of bytes.<br />
* The programmer reads "length" amount of bytes from the file, and uses this as the sought-after file data.<br />
<br />
==== ...performance fascist/programmer who wants low seek times ====<br />
* The programmer fseek()'s the file from the end until he finds 0x02014b50.<br />
* He then uses the following table to get the information he needs:<br />
{|class="wikitable"<br />
|+ ZIP central directory file header<br />
|-<br />
! Offset !! Bytes !! Description<br />
|-<br />
| 0 || 4 || Central directory file header signature = 0x02014b50<br />
|-<br />
| 4 || 2 || Version made by<br />
|-<br />
| 6 || 2 || Version needed to extract (minimum)<br />
|-<br />
| 8 || 2 || General purpose bit flag<br />
|-<br />
| 10 || 2 || Compression method<br />
|-<br />
| 12 || 2 || File last modification time<br />
|-<br />
| 14 || 2 || File last modification date<br />
|-<br />
| 16 || 4 || CRC-32<br />
|-<br />
| 20 || 4 || Compressed size<br />
|-<br />
| 24 || 4 || Uncompressed size<br />
|-<br />
| 28 || 2 || File name length (''n'')<br />
|-<br />
| 30 || 2 || Extra field length (''m'')<br />
|-<br />
| 32 || 2 || File comment length (''k'')<br />
|-<br />
| 34 || 2 || Disk number where file starts<br />
|-<br />
| 36 || 2 || Internal file attributes<br />
|-<br />
| 38 || 4 || External file attributes<br />
|-<br />
| 42 || 4 || Relative offset of local file header<br />
|-<br />
| 46 || ''n'' || File name<br />
|-<br />
| 46+''n'' || ''m'' || Extra field<br />
|-<br />
| 46+''n''+''m'' || ''k'' || File comment<br />
|}<br />
<br />
* The relative file offset can then be used to jump to the file in question. The file is then scanned as described in the previous section.<br />
<br />
== The 2014/Pyra effort ==<br />
<br />
A new format is being developed for the DragonBox Pyra handheld. Here are the requirements/notes:<br />
<br />
The compatibility layer should have its own table?<br />
<br />
{| class="wikitable"<br />
|-<br />
! Requirement !! Solutions !! Importance !! Status (decided, undecided, post-proned, won't fix, ...) !! Notes<br />
|-<br />
| Should the new format be supported by Pandora AND Pyra? ||<br />
* Yes<br />
** Makes releasing compatible applications easier.<br />
* no<br />
** Pandora is softfp and Pyra is hardfp - it's unlikely one package could work on both systems anyway.<br />
|| Very high || ||<br />
* http://boards.openpandora.org/topic/15530-the-software-side/page-7#entry310691<br />
* This decision affects several of the requirements and possible solutions on this table and should be decided sooner<br />
|-<br />
| API/Daemon to switch between joystick/keyboard mode || || || ||<br />
* Not exactly a PND-error<br />
* aTc gathered a nice list of notes: http://boards.openpandora.org/topic/15587-userspace-input-daemon/#entry311623<br />
|-<br />
| Guideline for consistent use of gaming buttons:<br />
* It should reduce the need of patches on top of the ported software<br />
||<br />
* A keyboard/joystick remapping function<br />
* A guideline for software developers<br />
|| || || No single-solution, but a set of them<br />
|-<br />
| Easier to parse than current PNDs and less fragile ||<br />
* header + metadata (UBJSON) + icon + screenshots + squashfs<br />
* tar archive<br />
** more future proof<br />
** easier for shell scripts<br />
*** with dd you can extract the tar part and so<br />
** filenames and file permissions are overhead<br />
*** But the overhead is too small to be significant?<br />
** easier to modify after-compiled<br />
*** not really a problem, because the user should be contributing to the recipe anyway<br />
*** a specialized tool can be create if another solution is chosen<br />
* zipfs uncompressed filesystem<br />
** See above sections<br />
* ar archive<br />
** Like .deb format<br />
|| || ||<br />
|-<br />
| Should the metadata include packages dependencies? ||<br />
* Yes<br />
** apt-get install package --print-uris will print download uris for dep packages. Using this uris download packages and extract all (dpkg -x 1.deb ./extracted)<br />
*** Not enough. It'll need two passes between an online and an offline system. It's impossible to be sure that an offline installation will work in one pass without including the whole repo: http://boards.openpandora.org/topic/15530-the-software-side/page-9#entry311602<br />
** EvilDragon voted for yes-and-no? http://boards.openpandora.org/topic/15530-the-software-side/page-5#entry310590<br />
*** Or does it mean a minimum "os-version required" metadata on the package and let the user update the system to have the latest libraries available?<br />
*** Possibly he doesn't care and just want something that just works: http://boards.openpandora.org/topic/15574-confliting-requirements/page-2#entry311580<br />
** http://boards.openpandora.org/topic/15530-the-software-side/page-5#entry310627<br />
* No.<br />
** EvilDragon commented that he wanna a plug-and-play-and-works system ( http://boards.openpandora.org/topic/15530-the-software-side/page-3#entry310412 )<br />
** If the user is lacking of internet access, "plug-and-play" package won't work<br />
** .deb for general-purpose software<br />
* set of dependencies that system is required to have (SDL, ...)<br />
* Complete user-space like containment features (see http://www.docker.io/ for an example)<br />
|| || Undecided ||<br />
* It is possible to tune the makepnd script to create flat PNDs and light PNDs.<br />
** Both packages would be included in the repo and user download what he likes more.<br />
* Independent of what choice is made, the minimum set of dependencies is a good idea to have around<br />
|-<br />
| It should be click-and-run ||<br />
* binfmt<br />
* Sheebang<br />
* File (extension) association<br />
|| || ||<br />
|-<br />
| Less duplication of metadata (c.f. PXML files) || || || ||<br />
|-<br />
| AUR-like system:<br />
* Building recipes is easier to replicate and contribute<br />
* If maintainer is jerk, moderator can move his maintanership<br />
|| Fork makepkg || || [http://boards.openpandora.org/topic/15530-the-software-side/page-7#entry310691 Cloudef is working on this task] ||<br />
|-<br />
| A compatibility layer for the current *.PND system:<br />
* The more transparent, the better<br />
||<br />
* Dual-boot<br />
* Virtual Machine<br />
* Tune Debian system to include all libraries used in current SuperZaxxon<br />
** OpenEmbedded it's based on the Debian, afterall<br />
** Probably won't happen because of hard/soft float point library issues<br />
* chroot<br />
** chroot doesn't start services like the real system, then the services of the chroot must be replaced/configured<br />
|| High || ||<br />
* Pyra will have larger internal space, then duplicate system isn't a large problem<br />
* For current OpenPandora, we can already run Debian on SD-card.<br />
** Maybe we can provide Pyra apps for OpenPandora using the Debian-on-sd-card opposed to Debian-on-internal-space<br />
|-<br />
| Cross-building? || || || Undecided ||<br />
|-<br />
| Home-dir of applications ||<br />
* Maybe one per SD-card<br />
* Union-fs? Aufs?<br />
* overridable by env var?<br />
|| || ||<br />
|-<br />
| Overridable scripts ||<br />
* By creating files with the right name, you can override files in the package (e.g. scripts)<br />
* Yes:<br />
** Cool for hackers<br />
* No:<br />
** Necessitates a union filesystem<br />
** Not absolutely necessary - hackers can just rebuild the package.<br />
|| || ||<br />
|-<br />
| File-extension? ||<br />
* .pyra<br />
** We're not using fucking DOS for lord-sake, what's the point of use 3-letters extension?<br />
* .pyd for pyra-data<br />
* .pyr<br />
* .pp for pyra program<br />
* .pa for pyra app<br />
* .drp for dragonbox pyra<br />
** Why limit this to just the Pyra? Could be useful in future systems/elsewhere too.<br />
* ...<br />
|| || undecided. || Maybe a series of polls on the boards? ;)<br />
|-<br />
| Keep what is good || Document all of the good features and port the content to this table || High || Not started ||<br />
|-<br />
| Extra bonus points for a design that can easily be used on other distros and devices as well || || Optional || Optional || Optional<br />
|-<br />
| Multiple binaries/icons per app || || || || Somebody (lost on the sea of comments) didn't like this design, but it should be possible for special software<br />
|-<br />
| Delta packages || || || || http://boards.openpandora.org/topic/15530-the-software-side/page-9#entry311600<br />
|-<br />
| PATH handling || || || || http://boards.openpandora.org/topic/15530-the-software-side/page-9#entry311618<br />
|}<br />
<br />
[[Category:PND]]<br />
[[Category:Ideas and proposals]]</div>Vinipsmakerhttps://pandorawiki.org/index.php?title=New_PND_format&diff=29478New PND format2014-02-06T04:39:04Z<p>Vinipsmaker: fix typo</p>
<hr />
<div>The current PND format has some shortcomings as listed below. This page should serve as a discussion page/white board for how the format could be improved.<br />
<br />
''' Proposal for discussion, this is just some opening shots '''<br />
<br />
== Current situation ==<br />
The current (ISO-based) PND format has the following shortcomings:<br />
<br />
* It currently uses ISO, SquashFS, and other filesystems... This means that there's no standard to follow, and libpnd needs to carry around support for a multitude of file systems.<br />
* It (often, but not always) uses the [http://en.wikipedia.org/wiki/ISO_9660 ISO file system] which is inefficient at storing data, because:<br />
** It contains duplicate headers for each file entry[http://users.telenet.be/it3.consultants.bvba/handouts/ISO9960.html] (one version with big-endian integers, one version with little-endian integers[http://en.wikipedia.org/wiki/Endianness]) which leads to many kilobytes of wasted space.<br />
** Its file tables are fixed-size, and there are therefore limitations on how many folders you can have, what names you can give your files, how big your files can be etc. For instance, files can only have names that are max 31 characters long, all upper-case, and limited to the ASCII character encoding, and an ISO does only support a folder depth of 8[http://www.on-time.com/rtos-32-docs/rtfiles-32/programming-manual/iso-9660/]. To support more, the Joliet file system extension is needed (or isofs won't recognize all file paths), and the Joliet header can be placed practically anywhere in the file, which means additional seek times.<br />
** It needs various file system extensions to behave correctly, like for example [http://en.wikipedia.org/wiki/Rock_Ridge Rock Ridge Interchange Protocol] and [http://en.wikipedia.org/wiki/Joliet_(file_system) Joliet]. Without them, it becomes pretty useless as demonstrated above, and with them, it becomes difficult to read by a tool (You can only use isofs! There are no other tools out there for programmers to use!).<br />
** It's very difficult to use since you need special ISO making tools to create the image. ISO is a relatively rare format if you're not technically inclined.<br />
** If you want to add or remove files from it, it's near impossible if you use a compact ISO file, since there's no way that you can "expand" an existing ISO file.<br />
* The PND header data is at the end (or in the middle of the file if a screenshot is included) which makes it impossible for e.g. libmagic to recognize the PND file. It will instead recognize it as an ISO file. Having the header data at the end also makes it take a very long time to find the data, making tools and the libpnd library very inefficient.<br />
* The PND file uses a custom XML format for its metadata. There's no reason to do this, especially since the established ".desktop" file format fills exactly the same function.<br />
(note: ".desktop" does not contain all metadata we have wanted over time, but it's of course possible to add extensions à la "X-Pandora-Whatever=xyz")<br />
* There's no "index" for the PND file. The whole file has to be scanned (albeit backwards) to find a PXML file, and there's a big chance for false positives etc.<br />
* Data is just appended linearly to the file so there's no order. If the format is to be extended (to e.g. include an icon file after the screenshot file), should the data just be appended as well?<br />
* UTF-8 is strictly the only encoding that is supported. If you make your PXML on a Windows machine, it won't work.<br />
<br />
== Proposed revisions ==<br />
=== Step 1: File system ===<br />
The file system for PNDs should be replaced by the ''uncompressed'' ZIP archive format. ZIP has the advantage that it's incredibly compact, and uncompressed ZIP makes it possible to read data from the file without having to do any decompression.<br />
<br />
ZIP files are mountable using various implementations of zipfs, and most of these implementations won't store files in memory when they are being read if the ZIP file is uncompressed.<br />
<br />
=== Step 2: Metadata ===<br />
PND files should no longer require special tools for them to be created. Therefore, since ZIP files support random access on files, it shouldn't be necessary to append/prepend metadata to the file. The user simply includes the "PXML.xml" and "preview.png" files inside of the ZIP, and any tools that need information about the PND can simply go to the central directory of the ZIP file (or use a simple ZIP library to do it for them) and get the location of the file inside of the ZIP. This should also dramatically decrease loading times for PNDs.<br />
<br />
Note; performance testing is needed; at the time we started, zip was enormously slower than plain ISO; with driver changes, it may or may not be so, hence our adopted multiple-filesystem-type system. zipfs is one possible option.<br />
<br />
=== Step 3: Metadata format ===<br />
Current PND files include so-called "PXML.xml" files. These files have a custom XML format that has a strange structure.<br />
<br />
These files should be replaced by ".desktop" files. A PND can contain one or more ".desktop" files in its root directory that specify how an application should be launched. PND tools simply use all ".desktop" files they can find in the PND when creating launchers for the contents of the PND.<br />
<br />
== Advantages ==<br />
* There don't have to be ''any'' special tools for reading PND files. The package can be run on any platform using any programming language that can read ZIP files.<br />
* We can use existing facilities to manage the launching of applications. The ".desktop" files can basically be copied without modification into standard locations of the system, and all launchers will become aware of them.<br />
* Reading PND files becomes easier (because of better tool support... sorry, but you can't link to libpnd in all programming languages) and quicker (becuase of the central directory for random file access).<br />
<br />
=== Benchmarks ===<br />
A benchmark was carried through to measure the performance of various package implementations.<br />
The compared systems were:<br />
* ZIP-based uncompressed packages using fuse-zip (a zipfs implementation) to read the files<br />
* ZIP-based compressed packages also using fuse-zip.<br />
* ISO8859-based packages using isofs<br />
* CramFS-based packages.<br />
<br />
A total of 9 files were generated for the test. They have sizes 1, 2, 3, 4, 16, 32, 64, 128 and 256 MB, and were generated via "dd if=/dev/urandom of=$file bs=1M count=$size".<br />
<br />
Because the files contain random data, they did not compress very well. The resulting files were:<br />
* "zipcompressed.zip" 511.1 MB<br />
* "zipuncompressed.zip" 511.0 MB<br />
* "iso.iso" 511.3 MB<br />
* "cramfs.image" 95.4 MB (WOW!)<br />
<br />
Some notes:<br />
* No write tests were done for obvious reasons<br />
* No random access tests were done since the Pandora will use SD cards and thus not get penalized from random access, and that's not what we're interested in anyways.<br />
* Before each test, the commands "sync; echo 3 > /proc/sys/vm/drop_caches" were run.<br />
<br />
==== First test: linear read time ====<br />
Command: "time cat mountpoint/* > /dev/null"<br />
<br />
* ISO: 8.119 seconds<br />
* CramFS: 1.488 seconds (WOW!)<br />
* Zip (compressed): 8.535 seconds<br />
* Zip (uncompressed): 8.290 seconds<br />
<br />
==== Second test: RAM usage ====<br />
-- TODO, have to find a good method of measuring this --<br />
<br />
== Remaining issues ==<br />
* How should libpnd tell the difference between old and new PND files? Should it depend on the "file" tool, should it run its own recognition algorithm, or what? Should the extension be changed to e.g. ".box" (Which was proposed in a thread)?<br />
* How to make sure that the ZIP files are uncompressed? Should we provide a script that "uncompresses" an ordinary ZIP file?<br />
* Is cramfs better than ZIP?<br />
** cramfs cannot be uncompressed like ZIP can<br />
** If you compare compressed ZIP and cramfs, cramfs is more efficient.<br />
** cramfs is harder to use than ZIP (for the developer).<br />
** cramfs ''needs'' to have all of its opened files in memory.<br />
<br />
== Upgrade path from the old PND format ==<br />
A simple script can be written that extracts the old PND file, including the screenshot and the PXML.xml file. The PXML.xml file is then converted to one or many .desktop files, and the desktop files, the preview.png file, and the package contents are compacted into a ZIP that then is renamed to "*.pnd"<br />
<br />
== Usage scenario ==<br />
=== Repackaging of an application from another package format ===<br />
* The user grabs the package for the application<br />
* He dumps the executable and all required libraries into a folder<br />
* He dumps a "screenshot.png" file into the folder that he's made<br />
* He copies the ".desktop" file(s) for the application from the old package, opens them, replaces "Exec=/usr/bin/bla" with "Exec=./bla" and saves them in the directory<br />
* He uses a ZIP archiver to make a ZIP out of the folder, making sure that he sets the "uncompressed" option.<br />
<br />
=== Creating PNDs as part of a build process ===<br />
* The build tool creates a directory with all of the necessary information like above and invokes the "zip" utility to compress the folder.<br />
<br />
=== Accessing a PND's contents for the... ===<br />
<br />
==== ...lazy programmer ====<br />
* The programmer extracts the PND into a folder via libzip and accesses its contents.<br />
<br />
==== ...pragmatic programmer ====<br />
* The programmer uses zipfs to mount the ZIP and accesses the file's contents.<br />
<br />
==== ...smart performance-aware programmer ====<br />
* The programmer fseek()'s the ZIP file until he finds 0x04034b50.<br />
* He jumps 18 bytes forward and reads the int at that location, storing it in "length".<br />
* He jumps 4 bytes and checks if this int matches "length". If it doesn't, it means that the file is compressed, and an error is reported.<br />
* He then jumps forward 4 bytes and stores the short at that location in a variable "nameLength".<br />
* Another 2-byte jump gets the short "extensionsLength".<br />
* He then jumps 2 bytes and reads "nameLength" amount of bytes from the file.<br />
* He then uses strcmp() to see if this string matches the sought-after file. If not, he continues the fseek().<br />
* The programmer now skips "entensionsLength" amount of bytes.<br />
* The programmer reads "length" amount of bytes from the file, and uses this as the sought-after file data.<br />
<br />
==== ...performance fascist/programmer who wants low seek times ====<br />
* The programmer fseek()'s the file from the end until he finds 0x02014b50.<br />
* He then uses the following table to get the information he needs:<br />
{|class="wikitable"<br />
|+ ZIP central directory file header<br />
|-<br />
! Offset !! Bytes !! Description<br />
|-<br />
| 0 || 4 || Central directory file header signature = 0x02014b50<br />
|-<br />
| 4 || 2 || Version made by<br />
|-<br />
| 6 || 2 || Version needed to extract (minimum)<br />
|-<br />
| 8 || 2 || General purpose bit flag<br />
|-<br />
| 10 || 2 || Compression method<br />
|-<br />
| 12 || 2 || File last modification time<br />
|-<br />
| 14 || 2 || File last modification date<br />
|-<br />
| 16 || 4 || CRC-32<br />
|-<br />
| 20 || 4 || Compressed size<br />
|-<br />
| 24 || 4 || Uncompressed size<br />
|-<br />
| 28 || 2 || File name length (''n'')<br />
|-<br />
| 30 || 2 || Extra field length (''m'')<br />
|-<br />
| 32 || 2 || File comment length (''k'')<br />
|-<br />
| 34 || 2 || Disk number where file starts<br />
|-<br />
| 36 || 2 || Internal file attributes<br />
|-<br />
| 38 || 4 || External file attributes<br />
|-<br />
| 42 || 4 || Relative offset of local file header<br />
|-<br />
| 46 || ''n'' || File name<br />
|-<br />
| 46+''n'' || ''m'' || Extra field<br />
|-<br />
| 46+''n''+''m'' || ''k'' || File comment<br />
|}<br />
<br />
* The relative file offset can then be used to jump to the file in question. The file is then scanned as described in the previous section.<br />
<br />
== The 2014/Pyra effort ==<br />
<br />
A new format is being developed for the DragonBox Pyra handheld. Here are the requirements/notes:<br />
<br />
The compatibility layer should have its own table?<br />
<br />
{| class="wikitable"<br />
|-<br />
! Requirement !! Solutions !! Importance !! Status (decided, undecided, post-proned, won't fix, ...) !! Notes<br />
|-<br />
| Should the new format be supported by Pandora AND Pyra? ||<br />
* Yes<br />
** Makes releasing compatible applications easier.<br />
* no<br />
** Pandora is softfp and Pyra is hardfp - it's unlikely one package could work on both systems anyway.<br />
|| Very high || ||<br />
* http://boards.openpandora.org/topic/15530-the-software-side/page-7#entry310691<br />
* This decision affects several of the requirements and possible solutions on this table and should be decided sooner<br />
|-<br />
| API/Daemon to switch between joystick/keyboard mode || || || ||<br />
* Not exactly a PND-error<br />
* aTc gathered a nice list of notes: http://boards.openpandora.org/topic/15587-userspace-input-daemon/#entry311623<br />
|-<br />
| Guideline for consistent use of gaming buttons:<br />
* It should reduce the need of patches on top of the ported software<br />
||<br />
* A keyboard/joystick remapping function<br />
* A guideline for software developers<br />
|| || || No single-solution, but a set of them<br />
|-<br />
| Easier to parse than current PNDs and less fragile ||<br />
* header + metadata (UBJSON) + icon + screenshots + squashfs<br />
* tar archive<br />
** more future proof<br />
** easier for shell scripts<br />
*** with dd you can extract the tar part and so<br />
** filenames and file permissions are overhead<br />
*** But the overhead is too small to be significant?<br />
** easier to modify after-compiled<br />
*** not really a problem, because the user should be contributing to the recipe anyway<br />
*** a specialized tool can be create if another solution is chosen<br />
* zipfs uncompressed filesystem<br />
** See above sections<br />
* ar archive<br />
** Like .deb format<br />
|| || ||<br />
|-<br />
| Should the metadata include packages dependencies? ||<br />
* Yes<br />
** apt-get install package --print-uris will print download uris for dep packages. Using this uris download packages and extract all (dpkg -x 1.deb ./extracted)<br />
*** Not enough. It'll need two passes between an online and an offline system. It's impossible to be sure that an offline installation will work in one pass without including the whole repo: http://boards.openpandora.org/topic/15530-the-software-side/page-9#entry311602<br />
** EvilDragon voted for yes-and-no? http://boards.openpandora.org/topic/15530-the-software-side/page-5#entry310590<br />
*** Or does it mean a minimum "os-version required" metadata on the package and let the user update the system to have the latest libraries available?<br />
*** Possibly he doesn't care and just want something that just works: http://boards.openpandora.org/topic/15574-confliting-requirements/page-2#entry311580<br />
** http://boards.openpandora.org/topic/15530-the-software-side/page-5#entry310627<br />
* No.<br />
** EvilDragon commented that he wanna a plug-and-play-and-works system ( http://boards.openpandora.org/topic/15530-the-software-side/page-3#entry310412 )<br />
** If the user is lacking of internet access, "plug-and-play" package won't work<br />
** .deb for general-purpose software<br />
* set of dependencies that system is required to have (SDL, ...)<br />
* Complete user-space like containment features (see http://www.docker.io/ for an example)<br />
|| || Undecided ||<br />
|-<br />
| It should be click-and-run ||<br />
* binfmt<br />
* Sheebang<br />
* File (extension) association<br />
|| || ||<br />
|-<br />
| Less duplication of metadata (c.f. PXML files) || || || ||<br />
|-<br />
| AUR-like system:<br />
* Building recipes is easier to replicate and contribute<br />
* If maintainer is jerk, moderator can move his maintanership<br />
|| Fork makepkg || || [http://boards.openpandora.org/topic/15530-the-software-side/page-7#entry310691 Cloudef is working on this task] ||<br />
|-<br />
| A compatibility layer for the current *.PND system:<br />
* The more transparent, the better<br />
||<br />
* Dual-boot<br />
* Virtual Machine<br />
* Tune Debian system to include all libraries used in current SuperZaxxon<br />
** OpenEmbedded it's based on the Debian, afterall<br />
** Probably won't happen because of hard/soft float point library issues<br />
* chroot<br />
** chroot doesn't start services like the real system, then the services of the chroot must be replaced/configured<br />
|| High || ||<br />
* Pyra will have larger internal space, then duplicate system isn't a large problem<br />
* For current OpenPandora, we can already run Debian on SD-card.<br />
** Maybe we can provide Pyra apps for OpenPandora using the Debian-on-sd-card opposed to Debian-on-internal-space<br />
|-<br />
| Cross-building? || || || Undecided ||<br />
|-<br />
| Home-dir of applications ||<br />
* Maybe one per SD-card<br />
* Union-fs? Aufs?<br />
* overridable by env var?<br />
|| || ||<br />
|-<br />
| Overridable scripts ||<br />
* By creating files with the right name, you can override files in the package (e.g. scripts)<br />
* Yes:<br />
** Cool for hackers<br />
* No:<br />
** Necessitates a union filesystem<br />
** Not absolutely necessary - hackers can just rebuild the package.<br />
|| || ||<br />
|-<br />
| File-extension? ||<br />
* .pyra<br />
** We're not using fucking DOS for lord-sake, what's the point of use 3-letters extension?<br />
* .pyd for pyra-data<br />
* .pyr<br />
* .pp for pyra program<br />
* .pa for pyra app<br />
* .drp for dragonbox pyra<br />
** Why limit this to just the Pyra? Could be useful in future systems/elsewhere too.<br />
* ...<br />
|| || undecided. || Maybe a series of polls on the boards? ;)<br />
|-<br />
| Keep what is good || Document all of the good features and port the content to this table || High || Not started ||<br />
|-<br />
| Extra bonus points for a design that can easily be used on other distros and devices as well || || Optional || Optional || Optional<br />
|-<br />
| Multiple binaries/icons per app || || || || Somebody (lost on the sea of comments) didn't like this design, but it should be possible for special software<br />
|-<br />
| Delta packages || || || || http://boards.openpandora.org/topic/15530-the-software-side/page-9#entry311600<br />
|-<br />
| PATH handling || || || || http://boards.openpandora.org/topic/15530-the-software-side/page-9#entry311618<br />
|}<br />
<br />
[[Category:PND]]<br />
[[Category:Ideas and proposals]]</div>Vinipsmakerhttps://pandorawiki.org/index.php?title=New_PND_format&diff=29477New PND format2014-02-06T04:36:01Z<p>Vinipsmaker: fix typo</p>
<hr />
<div>The current PND format has some shortcomings as listed below. This page should serve as a discussion page/white board for how the format could be improved.<br />
<br />
''' Proposal for discussion, this is just some opening shots '''<br />
<br />
== Current situation ==<br />
The current (ISO-based) PND format has the following shortcomings:<br />
<br />
* It currently uses ISO, SquashFS, and other filesystems... This means that there's no standard to follow, and libpnd needs to carry around support for a multitude of file systems.<br />
* It (often, but not always) uses the [http://en.wikipedia.org/wiki/ISO_9660 ISO file system] which is inefficient at storing data, because:<br />
** It contains duplicate headers for each file entry[http://users.telenet.be/it3.consultants.bvba/handouts/ISO9960.html] (one version with big-endian integers, one version with little-endian integers[http://en.wikipedia.org/wiki/Endianness]) which leads to many kilobytes of wasted space.<br />
** Its file tables are fixed-size, and there are therefore limitations on how many folders you can have, what names you can give your files, how big your files can be etc. For instance, files can only have names that are max 31 characters long, all upper-case, and limited to the ASCII character encoding, and an ISO does only support a folder depth of 8[http://www.on-time.com/rtos-32-docs/rtfiles-32/programming-manual/iso-9660/]. To support more, the Joliet file system extension is needed (or isofs won't recognize all file paths), and the Joliet header can be placed practically anywhere in the file, which means additional seek times.<br />
** It needs various file system extensions to behave correctly, like for example [http://en.wikipedia.org/wiki/Rock_Ridge Rock Ridge Interchange Protocol] and [http://en.wikipedia.org/wiki/Joliet_(file_system) Joliet]. Without them, it becomes pretty useless as demonstrated above, and with them, it becomes difficult to read by a tool (You can only use isofs! There are no other tools out there for programmers to use!).<br />
** It's very difficult to use since you need special ISO making tools to create the image. ISO is a relatively rare format if you're not technically inclined.<br />
** If you want to add or remove files from it, it's near impossible if you use a compact ISO file, since there's no way that you can "expand" an existing ISO file.<br />
* The PND header data is at the end (or in the middle of the file if a screenshot is included) which makes it impossible for e.g. libmagic to recognize the PND file. It will instead recognize it as an ISO file. Having the header data at the end also makes it take a very long time to find the data, making tools and the libpnd library very inefficient.<br />
* The PND file uses a custom XML format for its metadata. There's no reason to do this, especially since the established ".desktop" file format fills exactly the same function.<br />
(note: ".desktop" does not contain all metadata we have wanted over time, but it's of course possible to add extensions à la "X-Pandora-Whatever=xyz")<br />
* There's no "index" for the PND file. The whole file has to be scanned (albeit backwards) to find a PXML file, and there's a big chance for false positives etc.<br />
* Data is just appended linearly to the file so there's no order. If the format is to be extended (to e.g. include an icon file after the screenshot file), should the data just be appended as well?<br />
* UTF-8 is strictly the only encoding that is supported. If you make your PXML on a Windows machine, it won't work.<br />
<br />
== Proposed revisions ==<br />
=== Step 1: File system ===<br />
The file system for PNDs should be replaced by the ''uncompressed'' ZIP archive format. ZIP has the advantage that it's incredibly compact, and uncompressed ZIP makes it possible to read data from the file without having to do any decompression.<br />
<br />
ZIP files are mountable using various implementations of zipfs, and most of these implementations won't store files in memory when they are being read if the ZIP file is uncompressed.<br />
<br />
=== Step 2: Metadata ===<br />
PND files should no longer require special tools for them to be created. Therefore, since ZIP files support random access on files, it shouldn't be necessary to append/prepend metadata to the file. The user simply includes the "PXML.xml" and "preview.png" files inside of the ZIP, and any tools that need information about the PND can simply go to the central directory of the ZIP file (or use a simple ZIP library to do it for them) and get the location of the file inside of the ZIP. This should also dramatically decrease loading times for PNDs.<br />
<br />
Note; performance testing is needed; at the time we started, zip was enormously slower than plain ISO; with driver changes, it may or may not be so, hence our adopted multiple-filesystem-type system. zipfs is one possible option.<br />
<br />
=== Step 3: Metadata format ===<br />
Current PND files include so-called "PXML.xml" files. These files have a custom XML format that has a strange structure.<br />
<br />
These files should be replaced by ".desktop" files. A PND can contain one or more ".desktop" files in its root directory that specify how an application should be launched. PND tools simply use all ".desktop" files they can find in the PND when creating launchers for the contents of the PND.<br />
<br />
== Advantages ==<br />
* There don't have to be ''any'' special tools for reading PND files. The package can be run on any platform using any programming language that can read ZIP files.<br />
* We can use existing facilities to manage the launching of applications. The ".desktop" files can basically be copied without modification into standard locations of the system, and all launchers will become aware of them.<br />
* Reading PND files becomes easier (because of better tool support... sorry, but you can't link to libpnd in all programming languages) and quicker (becuase of the central directory for random file access).<br />
<br />
=== Benchmarks ===<br />
A benchmark was carried through to measure the performance of various package implementations.<br />
The compared systems were:<br />
* ZIP-based uncompressed packages using fuse-zip (a zipfs implementation) to read the files<br />
* ZIP-based compressed packages also using fuse-zip.<br />
* ISO8859-based packages using isofs<br />
* CramFS-based packages.<br />
<br />
A total of 9 files were generated for the test. They have sizes 1, 2, 3, 4, 16, 32, 64, 128 and 256 MB, and were generated via "dd if=/dev/urandom of=$file bs=1M count=$size".<br />
<br />
Because the files contain random data, they did not compress very well. The resulting files were:<br />
* "zipcompressed.zip" 511.1 MB<br />
* "zipuncompressed.zip" 511.0 MB<br />
* "iso.iso" 511.3 MB<br />
* "cramfs.image" 95.4 MB (WOW!)<br />
<br />
Some notes:<br />
* No write tests were done for obvious reasons<br />
* No random access tests were done since the Pandora will use SD cards and thus not get penalized from random access, and that's not what we're interested in anyways.<br />
* Before each test, the commands "sync; echo 3 > /proc/sys/vm/drop_caches" were run.<br />
<br />
==== First test: linear read time ====<br />
Command: "time cat mountpoint/* > /dev/null"<br />
<br />
* ISO: 8.119 seconds<br />
* CramFS: 1.488 seconds (WOW!)<br />
* Zip (compressed): 8.535 seconds<br />
* Zip (uncompressed): 8.290 seconds<br />
<br />
==== Second test: RAM usage ====<br />
-- TODO, have to find a good method of measuring this --<br />
<br />
== Remaining issues ==<br />
* How should libpnd tell the difference between old and new PND files? Should it depend on the "file" tool, should it run its own recognition algorithm, or what? Should the extension be changed to e.g. ".box" (Which was proposed in a thread)?<br />
* How to make sure that the ZIP files are uncompressed? Should we provide a script that "uncompresses" an ordinary ZIP file?<br />
* Is cramfs better than ZIP?<br />
** cramfs cannot be uncompressed like ZIP can<br />
** If you compare compressed ZIP and cramfs, cramfs is more efficient.<br />
** cramfs is harder to use than ZIP (for the developer).<br />
** cramfs ''needs'' to have all of its opened files in memory.<br />
<br />
== Upgrade path from the old PND format ==<br />
A simple script can be written that extracts the old PND file, including the screenshot and the PXML.xml file. The PXML.xml file is then converted to one or many .desktop files, and the desktop files, the preview.png file, and the package contents are compacted into a ZIP that then is renamed to "*.pnd"<br />
<br />
== Usage scenario ==<br />
=== Repackaging of an application from another package format ===<br />
* The user grabs the package for the application<br />
* He dumps the executable and all required libraries into a folder<br />
* He dumps a "screenshot.png" file into the folder that he's made<br />
* He copies the ".desktop" file(s) for the application from the old package, opens them, replaces "Exec=/usr/bin/bla" with "Exec=./bla" and saves them in the directory<br />
* He uses a ZIP archiver to make a ZIP out of the folder, making sure that he sets the "uncompressed" option.<br />
<br />
=== Creating PNDs as part of a build process ===<br />
* The build tool creates a directory with all of the necessary information like above and invokes the "zip" utility to compress the folder.<br />
<br />
=== Accessing a PND's contents for the... ===<br />
<br />
==== ...lazy programmer ====<br />
* The programmer extracts the PND into a folder via libzip and accesses its contents.<br />
<br />
==== ...pragmatic programmer ====<br />
* The programmer uses zipfs to mount the ZIP and accesses the file's contents.<br />
<br />
==== ...smart performance-aware programmer ====<br />
* The programmer fseek()'s the ZIP file until he finds 0x04034b50.<br />
* He jumps 18 bytes forward and reads the int at that location, storing it in "length".<br />
* He jumps 4 bytes and checks if this int matches "length". If it doesn't, it means that the file is compressed, and an error is reported.<br />
* He then jumps forward 4 bytes and stores the short at that location in a variable "nameLength".<br />
* Another 2-byte jump gets the short "extensionsLength".<br />
* He then jumps 2 bytes and reads "nameLength" amount of bytes from the file.<br />
* He then uses strcmp() to see if this string matches the sought-after file. If not, he continues the fseek().<br />
* The programmer now skips "entensionsLength" amount of bytes.<br />
* The programmer reads "length" amount of bytes from the file, and uses this as the sought-after file data.<br />
<br />
==== ...performance fascist/programmer who wants low seek times ====<br />
* The programmer fseek()'s the file from the end until he finds 0x02014b50.<br />
* He then uses the following table to get the information he needs:<br />
{|class="wikitable"<br />
|+ ZIP central directory file header<br />
|-<br />
! Offset !! Bytes !! Description<br />
|-<br />
| 0 || 4 || Central directory file header signature = 0x02014b50<br />
|-<br />
| 4 || 2 || Version made by<br />
|-<br />
| 6 || 2 || Version needed to extract (minimum)<br />
|-<br />
| 8 || 2 || General purpose bit flag<br />
|-<br />
| 10 || 2 || Compression method<br />
|-<br />
| 12 || 2 || File last modification time<br />
|-<br />
| 14 || 2 || File last modification date<br />
|-<br />
| 16 || 4 || CRC-32<br />
|-<br />
| 20 || 4 || Compressed size<br />
|-<br />
| 24 || 4 || Uncompressed size<br />
|-<br />
| 28 || 2 || File name length (''n'')<br />
|-<br />
| 30 || 2 || Extra field length (''m'')<br />
|-<br />
| 32 || 2 || File comment length (''k'')<br />
|-<br />
| 34 || 2 || Disk number where file starts<br />
|-<br />
| 36 || 2 || Internal file attributes<br />
|-<br />
| 38 || 4 || External file attributes<br />
|-<br />
| 42 || 4 || Relative offset of local file header<br />
|-<br />
| 46 || ''n'' || File name<br />
|-<br />
| 46+''n'' || ''m'' || Extra field<br />
|-<br />
| 46+''n''+''m'' || ''k'' || File comment<br />
|}<br />
<br />
* The relative file offset can then be used to jump to the file in question. The file is then scanned as described in the previous section.<br />
<br />
== The 2014/Pyra effort ==<br />
<br />
A new format is being developed for the DragonBox Pyra handheld. Here are the requirements/notes:<br />
<br />
The compatibility layer should have its own table?<br />
<br />
{| class="wikitable"<br />
|-<br />
! Requirement !! Solutions !! Importance !! Status (decided, undecided, post-proned, won't fix, ...) !! Notes<br />
|-<br />
| Should the new format be supported by Pandora AND Pyra? ||<br />
* Yes<br />
** Makes releasing compatible applications easier.<br />
* no<br />
** Pandora is softfp and Pyra is hardfp - it's unlikely one package could work on both systems anyway.<br />
|| Very high || ||<br />
* http://boards.openpandora.org/topic/15530-the-software-side/page-7#entry310691<br />
* This decision affects several of the requirements and possible solutions on this table and should be decided sooner<br />
|-<br />
| API/Daemon to switch between joystick/keyboard mode || || || ||<br />
* Not exactly a PND-error<br />
* aTc gathered a nice list of notes: http://boards.openpandora.org/topic/15587-userspace-input-daemon/#entry311623<br />
|-<br />
| Guideline for consistent use of gaming buttons:<br />
* It should reduce the need of patches on top of the ported software<br />
||<br />
* A keyboard/joystick remapping function<br />
* A guideline for software developers<br />
|| || || No single-solution, but a set of them<br />
|-<br />
| Easier to parse than current PNDs and less fragile ||<br />
* header + metadata (UBJSON) + icon + screenshots + squashfs<br />
* tar archive<br />
** more future proof<br />
** easier for shell scripts<br />
*** with dd you can extract the tar part and so<br />
** filenames and file permissions are overhead<br />
*** But the overhead is too small to be significant?<br />
** easier to modify after-compiled<br />
*** not really a problem, because the user should be contributing to the recipe anyway<br />
*** a specialized tool can be create if another solution is chosen<br />
* zipfs uncompressed filesystem<br />
** See above sections<br />
* ar archive<br />
** Like .deb format<br />
|| || ||<br />
|-<br />
| Should the metadata include packages dependencies? ||<br />
* Yes<br />
** apt-get install package --print-uris will print download uris for dep packages. Using this uris download packages and extract all (dpkg -x 1.deb ./extracted)<br />
*** Not enough. It'll need two passes between an online and an offline system. It's impossible to be sure that an offline installation will work in one pass without including the whole repo: http://boards.openpandora.org/topic/15530-the-software-side/page-9#entry311602<br />
** EvilDragon voted for yes-and-no? http://boards.openpandora.org/topic/15530-the-software-side/page-5#entry310590<br />
*** Or does it mean a minimum "os-version required" metadata on the package and let the user update the system to have the latest libraries available?<br />
*** Possibly he doesn't care and just want something that just works: http://boards.openpandora.org/topic/15574-confliting-requirements/page-2#entry311580<br />
** http://boards.openpandora.org/topic/15530-the-software-side/page-5#entry310627<br />
* No.<br />
** EvilDragon commented that he wanna a plug-and-play-and-works system ( http://boards.openpandora.org/topic/15530-the-software-side/page-3#entry310412 )<br />
** If the user is lacking of internet access, "plug-and-play" package won't work<br />
** .deb for general-purpose software<br />
* set of dependencies that system is required to have (SDL, ...)<br />
* Complete user-space like containment features (see http://www.docker.io/ for an example)<br />
|| || Undecided ||<br />
|-<br />
| It should be click-and-run ||<br />
* binfmt<br />
* Sheebang<br />
* File (extension) association<br />
|| || ||<br />
|-<br />
| Less duplication of metadata (c.f. PXML files) || || || ||<br />
|-<br />
| AUR-like system:<br />
* Building recipes is easier to replicate and contribute<br />
* If maintainer is jerk, moderator can move his maintanership<br />
|| Fork makepkg || || [http://boards.openpandora.org/topic/15530-the-software-side/page-7#entry310691 Cloudef is working on this task] ||<br />
|-<br />
| A compatibility layer for the current *.PND system:<br />
* The more transparent, the better<br />
||<br />
* Dual-boot<br />
* Virtual Machine<br />
* Tune Debian system to include all libraries used in current SuperZaxxon<br />
** OpenEmbedded it's based on the Debian, afterall<br />
** Probably won't happen because of hard/soft float point library issues<br />
* chroot<br />
** chroot doesn't start services like the real system, then the services of the chroot must be replaced/configured<br />
|| High || ||<br />
* Pyra will have larger internal space, then duplicate system isn't a large problem<br />
* For current OpenPandora, we can already run Debian on SD-card.<br />
** Maybe we can provide Pyra apps for OpenPandora using the Debian-on-sd-card opposed to Debian-on-internal-space<br />
|-<br />
| Cross-building? || || || Undecided ||<br />
|-<br />
| Home-dir of applications ||<br />
* Maybe on per SD-card<br />
* Union-fs? Aufs?<br />
* overridable by env var?<br />
|| || ||<br />
|-<br />
| Overridable scripts ||<br />
* By creating files with the right name, you can override files in the package (e.g. scripts)<br />
* Yes:<br />
** Cool for hackers<br />
* No:<br />
** Necessitates a union filesystem<br />
** Not absolutely necessary - hackers can just rebuild the package.<br />
|| || ||<br />
|-<br />
| File-extension? ||<br />
* .pyra<br />
** We're not using fucking DOS for lord-sake, what's the point of use 3-letters extension?<br />
* .pyd for pyra-data<br />
* .pyr<br />
* .pp for pyra program<br />
* .pa for pyra app<br />
* .drp for dragonbox pyra<br />
** Why limit this to just the Pyra? Could be useful in future systems/elsewhere too.<br />
* ...<br />
|| || undecided. || Maybe a series of polls on the boards? ;)<br />
|-<br />
| Keep what is good || Document all of the good features and port the content to this table || High || Not started ||<br />
|-<br />
| Extra bonus points for a design that can easily be used on other distros and devices as well || || Optional || Optional || Optional<br />
|-<br />
| Multiple binaries/icons per app || || || || Somebody (lost on the sea of comments) didn't like this design, but it should be possible for special software<br />
|-<br />
| Delta packages || || || || http://boards.openpandora.org/topic/15530-the-software-side/page-9#entry311600<br />
|-<br />
| PATH handling || || || || http://boards.openpandora.org/topic/15530-the-software-side/page-9#entry311618<br />
|}<br />
<br />
[[Category:PND]]<br />
[[Category:Ideas and proposals]]</div>Vinipsmakerhttps://pandorawiki.org/index.php?title=New_PND_format&diff=29475New PND format2014-02-05T21:51:40Z<p>Vinipsmaker: Reordered table to put related things closer</p>
<hr />
<div>The current PND format has some shortcomings as listed below. This page should serve as a discussion page/white board for how the format could be improved.<br />
<br />
''' Proposal for discussion, this is just some opening shots '''<br />
<br />
== Current situation ==<br />
The current (ISO-based) PND format has the following shortcomings:<br />
<br />
* It currently uses ISO, SquashFS, and other filesystems... This means that there's no standard to follow, and libpnd needs to carry around support for a multitude of file systems.<br />
* It (often, but not always) uses the [http://en.wikipedia.org/wiki/ISO_9660 ISO file system] which is inefficient at storing data, because:<br />
** It contains duplicate headers for each file entry[http://users.telenet.be/it3.consultants.bvba/handouts/ISO9960.html] (one version with big-endian integers, one version with little-endian integers[http://en.wikipedia.org/wiki/Endianness]) which leads to many kilobytes of wasted space.<br />
** Its file tables are fixed-size, and there are therefore limitations on how many folders you can have, what names you can give your files, how big your files can be etc. For instance, files can only have names that are max 31 characters long, all upper-case, and limited to the ASCII character encoding, and an ISO does only support a folder depth of 8[http://www.on-time.com/rtos-32-docs/rtfiles-32/programming-manual/iso-9660/]. To support more, the Joliet file system extension is needed (or isofs won't recognize all file paths), and the Joliet header can be placed practically anywhere in the file, which means additional seek times.<br />
** It needs various file system extensions to behave correctly, like for example [http://en.wikipedia.org/wiki/Rock_Ridge Rock Ridge Interchange Protocol] and [http://en.wikipedia.org/wiki/Joliet_(file_system) Joliet]. Without them, it becomes pretty useless as demonstrated above, and with them, it becomes difficult to read by a tool (You can only use isofs! There are no other tools out there for programmers to use!).<br />
** It's very difficult to use since you need special ISO making tools to create the image. ISO is a relatively rare format if you're not technically inclined.<br />
** If you want to add or remove files from it, it's near impossible if you use a compact ISO file, since there's no way that you can "expand" an existing ISO file.<br />
* The PND header data is at the end (or in the middle of the file if a screenshot is included) which makes it impossible for e.g. libmagic to recognize the PND file. It will instead recognize it as an ISO file. Having the header data at the end also makes it take a very long time to find the data, making tools and the libpnd library very inefficient.<br />
* The PND file uses a custom XML format for its metadata. There's no reason to do this, especially since the established ".desktop" file format fills exactly the same function.<br />
(note: ".desktop" does not contain all metadata we have wanted over time, but it's of course possible to add extensions à la "X-Pandora-Whatever=xyz")<br />
* There's no "index" for the PND file. The whole file has to be scanned (albeit backwards) to find a PXML file, and there's a big chance for false positives etc.<br />
* Data is just appended linearly to the file so there's no order. If the format is to be extended (to e.g. include an icon file after the screenshot file), should the data just be appended as well?<br />
* UTF-8 is strictly the only encoding that is supported. If you make your PXML on a Windows machine, it won't work.<br />
<br />
== Proposed revisions ==<br />
=== Step 1: File system ===<br />
The file system for PNDs should be replaced by the ''uncompressed'' ZIP archive format. ZIP has the advantage that it's incredibly compact, and uncompressed ZIP makes it possible to read data from the file without having to do any decompression.<br />
<br />
ZIP files are mountable using various implementations of zipfs, and most of these implementations won't store files in memory when they are being read if the ZIP file is uncompressed.<br />
<br />
=== Step 2: Metadata ===<br />
PND files should no longer require special tools for them to be created. Therefore, since ZIP files support random access on files, it shouldn't be necessary to append/prepend metadata to the file. The user simply includes the "PXML.xml" and "preview.png" files inside of the ZIP, and any tools that need information about the PND can simply go to the central directory of the ZIP file (or use a simple ZIP library to do it for them) and get the location of the file inside of the ZIP. This should also dramatically decrease loading times for PNDs.<br />
<br />
Note; performance testing is needed; at the time we started, zip was enormously slower than plain ISO; with driver changes, it may or may not be so, hence our adopted multiple-filesystem-type system. zipfs is one possible option.<br />
<br />
=== Step 3: Metadata format ===<br />
Current PND files include so-called "PXML.xml" files. These files have a custom XML format that has a strange structure.<br />
<br />
These files should be replaced by ".desktop" files. A PND can contain one or more ".desktop" files in its root directory that specify how an application should be launched. PND tools simply use all ".desktop" files they can find in the PND when creating launchers for the contents of the PND.<br />
<br />
== Advantages ==<br />
* There don't have to be ''any'' special tools for reading PND files. The package can be run on any platform using any programming language that can read ZIP files.<br />
* We can use existing facilities to manage the launching of applications. The ".desktop" files can basically be copied without modification into standard locations of the system, and all launchers will become aware of them.<br />
* Reading PND files becomes easier (because of better tool support... sorry, but you can't link to libpnd in all programming languages) and quicker (becuase of the central directory for random file access).<br />
<br />
=== Benchmarks ===<br />
A benchmark was carried through to measure the performance of various package implementations.<br />
The compared systems were:<br />
* ZIP-based uncompressed packages using fuse-zip (a zipfs implementation) to read the files<br />
* ZIP-based compressed packages also using fuse-zip.<br />
* ISO8859-based packages using isofs<br />
* CramFS-based packages.<br />
<br />
A total of 9 files were generated for the test. They have sizes 1, 2, 3, 4, 16, 32, 64, 128 and 256 MB, and were generated via "dd if=/dev/urandom of=$file bs=1M count=$size".<br />
<br />
Because the files contain random data, they did not compress very well. The resulting files were:<br />
* "zipcompressed.zip" 511.1 MB<br />
* "zipuncompressed.zip" 511.0 MB<br />
* "iso.iso" 511.3 MB<br />
* "cramfs.image" 95.4 MB (WOW!)<br />
<br />
Some notes:<br />
* No write tests were done for obvious reasons<br />
* No random access tests were done since the Pandora will use SD cards and thus not get penalized from random access, and that's not what we're interested in anyways.<br />
* Before each test, the commands "sync; echo 3 > /proc/sys/vm/drop_caches" were run.<br />
<br />
==== First test: linear read time ====<br />
Command: "time cat mountpoint/* > /dev/null"<br />
<br />
* ISO: 8.119 seconds<br />
* CramFS: 1.488 seconds (WOW!)<br />
* Zip (compressed): 8.535 seconds<br />
* Zip (uncompressed): 8.290 seconds<br />
<br />
==== Second test: RAM usage ====<br />
-- TODO, have to find a good method of measuring this --<br />
<br />
== Remaining issues ==<br />
* How should libpnd tell the difference between old and new PND files? Should it depend on the "file" tool, should it run its own recognition algorithm, or what? Should the extension be changed to e.g. ".box" (Which was proposed in a thread)?<br />
* How to make sure that the ZIP files are uncompressed? Should we provide a script that "uncompresses" an ordinary ZIP file?<br />
* Is cramfs better than ZIP?<br />
** cramfs cannot be uncompressed like ZIP can<br />
** If you compare compressed ZIP and cramfs, cramfs is more efficient.<br />
** cramfs is harder to use than ZIP (for the developer).<br />
** cramfs ''needs'' to have all of its opened files in memory.<br />
<br />
== Upgrade path from the old PND format ==<br />
A simple script can be written that extracts the old PND file, including the screenshot and the PXML.xml file. The PXML.xml file is then converted to one or many .desktop files, and the desktop files, the preview.png file, and the package contents are compacted into a ZIP that then is renamed to "*.pnd"<br />
<br />
== Usage scenario ==<br />
=== Repackaging of an application from another package format ===<br />
* The user grabs the package for the application<br />
* He dumps the executable and all required libraries into a folder<br />
* He dumps a "screenshot.png" file into the folder that he's made<br />
* He copies the ".desktop" file(s) for the application from the old package, opens them, replaces "Exec=/usr/bin/bla" with "Exec=./bla" and saves them in the directory<br />
* He uses a ZIP archiver to make a ZIP out of the folder, making sure that he sets the "uncompressed" option.<br />
<br />
=== Creating PNDs as part of a build process ===<br />
* The build tool creates a directory with all of the necessary information like above and invokes the "zip" utility to compress the folder.<br />
<br />
=== Accessing a PND's contents for the... ===<br />
<br />
==== ...lazy programmer ====<br />
* The programmer extracts the PND into a folder via libzip and accesses its contents.<br />
<br />
==== ...pragmatic programmer ====<br />
* The programmer uses zipfs to mount the ZIP and accesses the file's contents.<br />
<br />
==== ...smart performance-aware programmer ====<br />
* The programmer fseek()'s the ZIP file until he finds 0x04034b50.<br />
* He jumps 18 bytes forward and reads the int at that location, storing it in "length".<br />
* He jumps 4 bytes and checks if this int matches "length". If it doesn't, it means that the file is compressed, and an error is reported.<br />
* He then jumps forward 4 bytes and stores the short at that location in a variable "nameLength".<br />
* Another 2-byte jump gets the short "extensionsLength".<br />
* He then jumps 2 bytes and reads "nameLength" amount of bytes from the file.<br />
* He then uses strcmp() to see if this string matches the sought-after file. If not, he continues the fseek().<br />
* The programmer now skips "entensionsLength" amount of bytes.<br />
* The programmer reads "length" amount of bytes from the file, and uses this as the sought-after file data.<br />
<br />
==== ...performance fascist/programmer who wants low seek times ====<br />
* The programmer fseek()'s the file from the end until he finds 0x02014b50.<br />
* He then uses the following table to get the information he needs:<br />
{|class="wikitable"<br />
|+ ZIP central directory file header<br />
|-<br />
! Offset !! Bytes !! Description<br />
|-<br />
| 0 || 4 || Central directory file header signature = 0x02014b50<br />
|-<br />
| 4 || 2 || Version made by<br />
|-<br />
| 6 || 2 || Version needed to extract (minimum)<br />
|-<br />
| 8 || 2 || General purpose bit flag<br />
|-<br />
| 10 || 2 || Compression method<br />
|-<br />
| 12 || 2 || File last modification time<br />
|-<br />
| 14 || 2 || File last modification date<br />
|-<br />
| 16 || 4 || CRC-32<br />
|-<br />
| 20 || 4 || Compressed size<br />
|-<br />
| 24 || 4 || Uncompressed size<br />
|-<br />
| 28 || 2 || File name length (''n'')<br />
|-<br />
| 30 || 2 || Extra field length (''m'')<br />
|-<br />
| 32 || 2 || File comment length (''k'')<br />
|-<br />
| 34 || 2 || Disk number where file starts<br />
|-<br />
| 36 || 2 || Internal file attributes<br />
|-<br />
| 38 || 4 || External file attributes<br />
|-<br />
| 42 || 4 || Relative offset of local file header<br />
|-<br />
| 46 || ''n'' || File name<br />
|-<br />
| 46+''n'' || ''m'' || Extra field<br />
|-<br />
| 46+''n''+''m'' || ''k'' || File comment<br />
|}<br />
<br />
* The relative file offset can then be used to jump to the file in question. The file is then scanned as described in the previous section.<br />
<br />
== The 2014/Pyra effort ==<br />
<br />
A new format is being developed for the DragonBox Pyra handheld. Here are the requirements/notes:<br />
<br />
The compatibility layer should have its own table?<br />
<br />
{| class="wikitable"<br />
|-<br />
! Requirement !! Solutions !! Importance !! Status (decided, undecided, post-proned, won't fix, ...) !! Notes<br />
|-<br />
| Should the new format be supported by Pandora AND Pyra? ||<br />
* Yes<br />
* no<br />
|| Very high || ||<br />
* http://boards.openpandora.org/topic/15530-the-software-side/page-7#entry310691<br />
* This decision affects several of the requirements and possible solutions on this table and should be decided sooner<br />
|-<br />
| API/Daemon to switch between joystick/keyboard mode || || || ||<br />
* Not exactly a PND-error<br />
* aTc gathered a nice list of notes: http://boards.openpandora.org/topic/15587-userspace-input-daemon/#entry311623<br />
|-<br />
| Guideline for consistent use of gaming buttons:<br />
* It should reduce the need of patches on top of the ported software<br />
||<br />
* A keyboard/joystick remapping function<br />
* A guideline for software developers<br />
|| || || No single-solution, but a set of them<br />
|-<br />
| Easier to parse than current PNDs and less fragile ||<br />
* UBSON header and metadata + icon + screenshots + cramfs<br />
* tar filesystem<br />
** more future proof<br />
** easier for shell scripts<br />
*** with dd you can extract the tar part and so<br />
** filenames and file permissions are overhead<br />
*** But the overhead is too small to be significant?<br />
** easier to modify after-compiled<br />
*** not really a problem, because the user should be contributing to the recipe anyway<br />
*** a specialized to can be create if another solution is chosen<br />
* zipfs uncompressed filesystem<br />
** See above sections<br />
|| || ||<br />
|-<br />
| Should the metadata include packages dependencies? ||<br />
* Yes<br />
** apt-get install package --print-uris will print download uris for dep packages. Using this uris download packages and extract all (dpkg -x 1.deb ./extracted)<br />
*** Not enough. It'll need two passes between an online and an offline system. It's impossible to be sure that an offline installation will work in one pass without including the whole repo: http://boards.openpandora.org/topic/15530-the-software-side/page-9#entry311602<br />
** EvilDragon voted for yes-and-no? http://boards.openpandora.org/topic/15530-the-software-side/page-5#entry310590<br />
*** Or does it mean a minimum "os-version required" metadata on the package and let the user update the system to have the latest libraries available?<br />
*** Possibly he doesn't care and just want something that just works: http://boards.openpandora.org/topic/15574-confliting-requirements/page-2#entry311580<br />
** http://boards.openpandora.org/topic/15530-the-software-side/page-5#entry310627<br />
* No.<br />
** EvilDragon commented that he wanna a plug-and-play-and-works system ( http://boards.openpandora.org/topic/15530-the-software-side/page-3#entry310412 )<br />
** If the user is lacking of internet access, "plug-and-play" package won't work<br />
** .deb for general-purpose software<br />
* set of dependencies that system is required to have (SDL, ...)<br />
* Complete user-space like containment features (see http://www.docker.io/ for an example)<br />
|| || Undecided ||<br />
|-<br />
| It should be click-and-run ||<br />
* binfmt<br />
* Sheebang<br />
* File (extension) association<br />
|| || ||<br />
|-<br />
| Less duplication of .desktop and PXML files || || || ||<br />
|-<br />
| AUR-like system:<br />
* Building recipes is easier to replicate and contribute<br />
* If maintainer is jerk, moderator can move his maintanership<br />
|| Fork makepkg || || [http://boards.openpandora.org/topic/15530-the-software-side/page-7#entry310691 Cloudef is working on this task] ||<br />
|-<br />
| A compatibility layer for the current *.PND system:<br />
* The more transparent, the better<br />
||<br />
* Dual-boot<br />
* Virtual Machine<br />
* Tune Debian system to include all libraries used in current SuperZaxxon<br />
** OpenEmbedded it's based on the Debian, afterall<br />
** Probably won't happen because of hard/soft float point library issues<br />
* chroot<br />
** chroot doesn't start services like the real system, then the services of the chroot must be replaced/configured<br />
|| High || ||<br />
* Pyra will have larger internal space, then duplicate system isn't a large problem<br />
* For current OpenPandora, we can already run Debian on SD-card.<br />
** Maybe we can provide Pyra apps for OpenPandora using the Debian-on-sd-card opposed to Debian-on-internal-space<br />
|-<br />
| Cross-building? || || || Undecided ||<br />
|-<br />
| Filesystem type ||<br />
* Cramfs<br />
* Squashfilesystem<br />
|| || ||<br />
|-<br />
| Home-dir of applications ||<br />
* Maybe on per SD-card<br />
* Union-fs?<br />
* overridable by env var?<br />
|| || ||<br />
|-<br />
| Overridable scripts ||<br />
* By creating files with the right name, you can override files in the package (e.g. scripts)<br />
* Yes:<br />
** Cool for hackers<br />
* No:<br />
** Necessitates unionFS.<br />
** Not absolutely necessary - hackers can just rebuild the package.<br />
|| || ||<br />
|-<br />
| File-extension? ||<br />
* .pyra<br />
** We're not using fucking DOS for lord-sake, what's the point of use 3-letters extension?<br />
* .pyd for pyra-data<br />
* .pyr<br />
* .pp for pyra program<br />
* .pa for pyra app<br />
* .drp for dragonbox pyra<br />
** Why limit this to just the Pyra? Could be useful in future systems/elsewhere too.<br />
* ...<br />
|| || undecided. || Maybe a series of polls on the boards?<br />
|-<br />
| Keep what is good || Document all of the good features and port the content to this table || High || Not started ||<br />
|-<br />
| Extra bonus points for a design that can easily be used on other distros and devices as well || || Optional || Optional || Optional<br />
|-<br />
| Multiple binaries/icons per app || || || || Somebody (lost on the sea of comments) didn't like this design, but it should be possible for special software<br />
|-<br />
| Delta packages || || || || http://boards.openpandora.org/topic/15530-the-software-side/page-9#entry311600<br />
|-<br />
| PATH handling || || || || http://boards.openpandora.org/topic/15530-the-software-side/page-9#entry311618<br />
|}<br />
<br />
[[Category:PND]]<br />
[[Category:Ideas and proposals]]</div>Vinipsmakerhttps://pandorawiki.org/index.php?title=New_PND_format&diff=29474New PND format2014-02-05T21:49:24Z<p>Vinipsmaker: clarification of joystick/keyboard mode</p>
<hr />
<div>The current PND format has some shortcomings as listed below. This page should serve as a discussion page/white board for how the format could be improved.<br />
<br />
''' Proposal for discussion, this is just some opening shots '''<br />
<br />
== Current situation ==<br />
The current (ISO-based) PND format has the following shortcomings:<br />
<br />
* It currently uses ISO, SquashFS, and other filesystems... This means that there's no standard to follow, and libpnd needs to carry around support for a multitude of file systems.<br />
* It (often, but not always) uses the [http://en.wikipedia.org/wiki/ISO_9660 ISO file system] which is inefficient at storing data, because:<br />
** It contains duplicate headers for each file entry[http://users.telenet.be/it3.consultants.bvba/handouts/ISO9960.html] (one version with big-endian integers, one version with little-endian integers[http://en.wikipedia.org/wiki/Endianness]) which leads to many kilobytes of wasted space.<br />
** Its file tables are fixed-size, and there are therefore limitations on how many folders you can have, what names you can give your files, how big your files can be etc. For instance, files can only have names that are max 31 characters long, all upper-case, and limited to the ASCII character encoding, and an ISO does only support a folder depth of 8[http://www.on-time.com/rtos-32-docs/rtfiles-32/programming-manual/iso-9660/]. To support more, the Joliet file system extension is needed (or isofs won't recognize all file paths), and the Joliet header can be placed practically anywhere in the file, which means additional seek times.<br />
** It needs various file system extensions to behave correctly, like for example [http://en.wikipedia.org/wiki/Rock_Ridge Rock Ridge Interchange Protocol] and [http://en.wikipedia.org/wiki/Joliet_(file_system) Joliet]. Without them, it becomes pretty useless as demonstrated above, and with them, it becomes difficult to read by a tool (You can only use isofs! There are no other tools out there for programmers to use!).<br />
** It's very difficult to use since you need special ISO making tools to create the image. ISO is a relatively rare format if you're not technically inclined.<br />
** If you want to add or remove files from it, it's near impossible if you use a compact ISO file, since there's no way that you can "expand" an existing ISO file.<br />
* The PND header data is at the end (or in the middle of the file if a screenshot is included) which makes it impossible for e.g. libmagic to recognize the PND file. It will instead recognize it as an ISO file. Having the header data at the end also makes it take a very long time to find the data, making tools and the libpnd library very inefficient.<br />
* The PND file uses a custom XML format for its metadata. There's no reason to do this, especially since the established ".desktop" file format fills exactly the same function.<br />
(note: ".desktop" does not contain all metadata we have wanted over time, but it's of course possible to add extensions à la "X-Pandora-Whatever=xyz")<br />
* There's no "index" for the PND file. The whole file has to be scanned (albeit backwards) to find a PXML file, and there's a big chance for false positives etc.<br />
* Data is just appended linearly to the file so there's no order. If the format is to be extended (to e.g. include an icon file after the screenshot file), should the data just be appended as well?<br />
* UTF-8 is strictly the only encoding that is supported. If you make your PXML on a Windows machine, it won't work.<br />
<br />
== Proposed revisions ==<br />
=== Step 1: File system ===<br />
The file system for PNDs should be replaced by the ''uncompressed'' ZIP archive format. ZIP has the advantage that it's incredibly compact, and uncompressed ZIP makes it possible to read data from the file without having to do any decompression.<br />
<br />
ZIP files are mountable using various implementations of zipfs, and most of these implementations won't store files in memory when they are being read if the ZIP file is uncompressed.<br />
<br />
=== Step 2: Metadata ===<br />
PND files should no longer require special tools for them to be created. Therefore, since ZIP files support random access on files, it shouldn't be necessary to append/prepend metadata to the file. The user simply includes the "PXML.xml" and "preview.png" files inside of the ZIP, and any tools that need information about the PND can simply go to the central directory of the ZIP file (or use a simple ZIP library to do it for them) and get the location of the file inside of the ZIP. This should also dramatically decrease loading times for PNDs.<br />
<br />
Note; performance testing is needed; at the time we started, zip was enormously slower than plain ISO; with driver changes, it may or may not be so, hence our adopted multiple-filesystem-type system. zipfs is one possible option.<br />
<br />
=== Step 3: Metadata format ===<br />
Current PND files include so-called "PXML.xml" files. These files have a custom XML format that has a strange structure.<br />
<br />
These files should be replaced by ".desktop" files. A PND can contain one or more ".desktop" files in its root directory that specify how an application should be launched. PND tools simply use all ".desktop" files they can find in the PND when creating launchers for the contents of the PND.<br />
<br />
== Advantages ==<br />
* There don't have to be ''any'' special tools for reading PND files. The package can be run on any platform using any programming language that can read ZIP files.<br />
* We can use existing facilities to manage the launching of applications. The ".desktop" files can basically be copied without modification into standard locations of the system, and all launchers will become aware of them.<br />
* Reading PND files becomes easier (because of better tool support... sorry, but you can't link to libpnd in all programming languages) and quicker (becuase of the central directory for random file access).<br />
<br />
=== Benchmarks ===<br />
A benchmark was carried through to measure the performance of various package implementations.<br />
The compared systems were:<br />
* ZIP-based uncompressed packages using fuse-zip (a zipfs implementation) to read the files<br />
* ZIP-based compressed packages also using fuse-zip.<br />
* ISO8859-based packages using isofs<br />
* CramFS-based packages.<br />
<br />
A total of 9 files were generated for the test. They have sizes 1, 2, 3, 4, 16, 32, 64, 128 and 256 MB, and were generated via "dd if=/dev/urandom of=$file bs=1M count=$size".<br />
<br />
Because the files contain random data, they did not compress very well. The resulting files were:<br />
* "zipcompressed.zip" 511.1 MB<br />
* "zipuncompressed.zip" 511.0 MB<br />
* "iso.iso" 511.3 MB<br />
* "cramfs.image" 95.4 MB (WOW!)<br />
<br />
Some notes:<br />
* No write tests were done for obvious reasons<br />
* No random access tests were done since the Pandora will use SD cards and thus not get penalized from random access, and that's not what we're interested in anyways.<br />
* Before each test, the commands "sync; echo 3 > /proc/sys/vm/drop_caches" were run.<br />
<br />
==== First test: linear read time ====<br />
Command: "time cat mountpoint/* > /dev/null"<br />
<br />
* ISO: 8.119 seconds<br />
* CramFS: 1.488 seconds (WOW!)<br />
* Zip (compressed): 8.535 seconds<br />
* Zip (uncompressed): 8.290 seconds<br />
<br />
==== Second test: RAM usage ====<br />
-- TODO, have to find a good method of measuring this --<br />
<br />
== Remaining issues ==<br />
* How should libpnd tell the difference between old and new PND files? Should it depend on the "file" tool, should it run its own recognition algorithm, or what? Should the extension be changed to e.g. ".box" (Which was proposed in a thread)?<br />
* How to make sure that the ZIP files are uncompressed? Should we provide a script that "uncompresses" an ordinary ZIP file?<br />
* Is cramfs better than ZIP?<br />
** cramfs cannot be uncompressed like ZIP can<br />
** If you compare compressed ZIP and cramfs, cramfs is more efficient.<br />
** cramfs is harder to use than ZIP (for the developer).<br />
** cramfs ''needs'' to have all of its opened files in memory.<br />
<br />
== Upgrade path from the old PND format ==<br />
A simple script can be written that extracts the old PND file, including the screenshot and the PXML.xml file. The PXML.xml file is then converted to one or many .desktop files, and the desktop files, the preview.png file, and the package contents are compacted into a ZIP that then is renamed to "*.pnd"<br />
<br />
== Usage scenario ==<br />
=== Repackaging of an application from another package format ===<br />
* The user grabs the package for the application<br />
* He dumps the executable and all required libraries into a folder<br />
* He dumps a "screenshot.png" file into the folder that he's made<br />
* He copies the ".desktop" file(s) for the application from the old package, opens them, replaces "Exec=/usr/bin/bla" with "Exec=./bla" and saves them in the directory<br />
* He uses a ZIP archiver to make a ZIP out of the folder, making sure that he sets the "uncompressed" option.<br />
<br />
=== Creating PNDs as part of a build process ===<br />
* The build tool creates a directory with all of the necessary information like above and invokes the "zip" utility to compress the folder.<br />
<br />
=== Accessing a PND's contents for the... ===<br />
<br />
==== ...lazy programmer ====<br />
* The programmer extracts the PND into a folder via libzip and accesses its contents.<br />
<br />
==== ...pragmatic programmer ====<br />
* The programmer uses zipfs to mount the ZIP and accesses the file's contents.<br />
<br />
==== ...smart performance-aware programmer ====<br />
* The programmer fseek()'s the ZIP file until he finds 0x04034b50.<br />
* He jumps 18 bytes forward and reads the int at that location, storing it in "length".<br />
* He jumps 4 bytes and checks if this int matches "length". If it doesn't, it means that the file is compressed, and an error is reported.<br />
* He then jumps forward 4 bytes and stores the short at that location in a variable "nameLength".<br />
* Another 2-byte jump gets the short "extensionsLength".<br />
* He then jumps 2 bytes and reads "nameLength" amount of bytes from the file.<br />
* He then uses strcmp() to see if this string matches the sought-after file. If not, he continues the fseek().<br />
* The programmer now skips "entensionsLength" amount of bytes.<br />
* The programmer reads "length" amount of bytes from the file, and uses this as the sought-after file data.<br />
<br />
==== ...performance fascist/programmer who wants low seek times ====<br />
* The programmer fseek()'s the file from the end until he finds 0x02014b50.<br />
* He then uses the following table to get the information he needs:<br />
{|class="wikitable"<br />
|+ ZIP central directory file header<br />
|-<br />
! Offset !! Bytes !! Description<br />
|-<br />
| 0 || 4 || Central directory file header signature = 0x02014b50<br />
|-<br />
| 4 || 2 || Version made by<br />
|-<br />
| 6 || 2 || Version needed to extract (minimum)<br />
|-<br />
| 8 || 2 || General purpose bit flag<br />
|-<br />
| 10 || 2 || Compression method<br />
|-<br />
| 12 || 2 || File last modification time<br />
|-<br />
| 14 || 2 || File last modification date<br />
|-<br />
| 16 || 4 || CRC-32<br />
|-<br />
| 20 || 4 || Compressed size<br />
|-<br />
| 24 || 4 || Uncompressed size<br />
|-<br />
| 28 || 2 || File name length (''n'')<br />
|-<br />
| 30 || 2 || Extra field length (''m'')<br />
|-<br />
| 32 || 2 || File comment length (''k'')<br />
|-<br />
| 34 || 2 || Disk number where file starts<br />
|-<br />
| 36 || 2 || Internal file attributes<br />
|-<br />
| 38 || 4 || External file attributes<br />
|-<br />
| 42 || 4 || Relative offset of local file header<br />
|-<br />
| 46 || ''n'' || File name<br />
|-<br />
| 46+''n'' || ''m'' || Extra field<br />
|-<br />
| 46+''n''+''m'' || ''k'' || File comment<br />
|}<br />
<br />
* The relative file offset can then be used to jump to the file in question. The file is then scanned as described in the previous section.<br />
<br />
== The 2014/Pyra effort ==<br />
<br />
A new format is being developed for the DragonBox Pyra handheld. Here are the requirements/notes:<br />
<br />
The compatibility layer should have its own table?<br />
<br />
{| class="wikitable"<br />
|-<br />
! Requirement !! Solutions !! Importance !! Status (decided, undecided, post-proned, won't fix, ...) !! Notes<br />
|-<br />
| Should the new format be supported by Pandora AND Pyra? ||<br />
* Yes<br />
* no<br />
|| Very high || ||<br />
* http://boards.openpandora.org/topic/15530-the-software-side/page-7#entry310691<br />
* This decision affects several of the requirements and possible solutions on this table and should be decided sooner<br />
|-<br />
| API/Daemon to switch between joystick/keyboard mode || || || ||<br />
* Not exactly a PND-error<br />
* aTc gathered a nice list of requirements: http://boards.openpandora.org/topic/15587-userspace-input-daemon/#entry311623<br />
|-<br />
| Easier to parse than current PNDs and less fragile ||<br />
* UBSON header and metadata + icon + screenshots + cramfs<br />
* tar filesystem<br />
** more future proof<br />
** easier for shell scripts<br />
*** with dd you can extract the tar part and so<br />
** filenames and file permissions are overhead<br />
*** But the overhead is too small to be significant?<br />
** easier to modify after-compiled<br />
*** not really a problem, because the user should be contributing to the recipe anyway<br />
*** a specialized to can be create if another solution is chosen<br />
* zipfs uncompressed filesystem<br />
** See above sections<br />
|| || ||<br />
|-<br />
| Should the metadata include packages dependencies? ||<br />
* Yes<br />
** apt-get install package --print-uris will print download uris for dep packages. Using this uris download packages and extract all (dpkg -x 1.deb ./extracted)<br />
*** Not enough. It'll need two passes between an online and an offline system. It's impossible to be sure that an offline installation will work in one pass without including the whole repo: http://boards.openpandora.org/topic/15530-the-software-side/page-9#entry311602<br />
** EvilDragon voted for yes-and-no? http://boards.openpandora.org/topic/15530-the-software-side/page-5#entry310590<br />
*** Or does it mean a minimum "os-version required" metadata on the package and let the user update the system to have the latest libraries available?<br />
*** Possibly he doesn't care and just want something that just works: http://boards.openpandora.org/topic/15574-confliting-requirements/page-2#entry311580<br />
** http://boards.openpandora.org/topic/15530-the-software-side/page-5#entry310627<br />
* No.<br />
** EvilDragon commented that he wanna a plug-and-play-and-works system ( http://boards.openpandora.org/topic/15530-the-software-side/page-3#entry310412 )<br />
** If the user is lacking of internet access, "plug-and-play" package won't work<br />
** .deb for general-purpose software<br />
* set of dependencies that system is required to have (SDL, ...)<br />
* Complete user-space like containment features (see http://www.docker.io/ for an example)<br />
|| || Undecided ||<br />
|-<br />
| It should be click-and-run ||<br />
* binfmt<br />
* Sheebang<br />
* File (extension) association<br />
|| || ||<br />
|-<br />
| Less duplication of .desktop and PXML files || || || ||<br />
|-<br />
| AUR-like system:<br />
* Building recipes is easier to replicate and contribute<br />
* If maintainer is jerk, moderator can move his maintanership<br />
|| Fork makepkg || || [http://boards.openpandora.org/topic/15530-the-software-side/page-7#entry310691 Cloudef is working on this task] ||<br />
|-<br />
| Guideline for consistent use of gaming buttons:<br />
* It should reduce the need of patches on top of the ported software<br />
||<br />
* A keyboard/joystick remapping function<br />
* A guideline for software developers<br />
|| || || No single-solution, but a set of them<br />
|-<br />
| A compatibility layer for the current *.PND system:<br />
* The more transparent, the better<br />
||<br />
* Dual-boot<br />
* Virtual Machine<br />
* Tune Debian system to include all libraries used in current SuperZaxxon<br />
** OpenEmbedded it's based on the Debian, afterall<br />
** Probably won't happen because of hard/soft float point library issues<br />
* chroot<br />
** chroot doesn't start services like the real system, then the services of the chroot must be replaced/configured<br />
|| High || ||<br />
* Pyra will have larger internal space, then duplicate system isn't a large problem<br />
* For current OpenPandora, we can already run Debian on SD-card.<br />
** Maybe we can provide Pyra apps for OpenPandora using the Debian-on-sd-card opposed to Debian-on-internal-space<br />
|-<br />
| Cross-building? || || || Undecided ||<br />
|-<br />
| Filesystem type ||<br />
* Cramfs<br />
* Squashfilesystem<br />
|| || ||<br />
|-<br />
| Home-dir of applications ||<br />
* Maybe on per SD-card<br />
* Union-fs?<br />
* overridable by env var?<br />
|| || ||<br />
|-<br />
| Overridable scripts ||<br />
* By creating files with the right name, you can override files in the package (e.g. scripts)<br />
* Yes:<br />
** Cool for hackers<br />
* No:<br />
** Necessitates unionFS.<br />
** Not absolutely necessary - hackers can just rebuild the package.<br />
|| || ||<br />
|-<br />
| File-extension? ||<br />
* .pyra<br />
** We're not using fucking DOS for lord-sake, what's the point of use 3-letters extension?<br />
* .pyd for pyra-data<br />
* .pyr<br />
* .pp for pyra program<br />
* .pa for pyra app<br />
* .drp for dragonbox pyra<br />
** Why limit this to just the Pyra? Could be useful in future systems/elsewhere too.<br />
* ...<br />
|| || undecided. || Maybe a series of polls on the boards?<br />
|-<br />
| Keep what is good || Document all of the good features and port the content to this table || High || Not started ||<br />
|-<br />
| Extra bonus points for a design that can easily be used on other distros and devices as well || || Optional || Optional || Optional<br />
|-<br />
| Multiple binaries/icons per app || || || || Somebody (lost on the sea of comments) didn't like this design, but it should be possible for special software<br />
|-<br />
| Delta packages || || || || http://boards.openpandora.org/topic/15530-the-software-side/page-9#entry311600<br />
|-<br />
| PATH handling || || || || http://boards.openpandora.org/topic/15530-the-software-side/page-9#entry311618<br />
|}<br />
<br />
[[Category:PND]]<br />
[[Category:Ideas and proposals]]</div>Vinipsmakerhttps://pandorawiki.org/index.php?title=New_PND_format&diff=29473New PND format2014-02-05T21:47:03Z<p>Vinipsmaker: clarification of deps</p>
<hr />
<div>The current PND format has some shortcomings as listed below. This page should serve as a discussion page/white board for how the format could be improved.<br />
<br />
''' Proposal for discussion, this is just some opening shots '''<br />
<br />
== Current situation ==<br />
The current (ISO-based) PND format has the following shortcomings:<br />
<br />
* It currently uses ISO, SquashFS, and other filesystems... This means that there's no standard to follow, and libpnd needs to carry around support for a multitude of file systems.<br />
* It (often, but not always) uses the [http://en.wikipedia.org/wiki/ISO_9660 ISO file system] which is inefficient at storing data, because:<br />
** It contains duplicate headers for each file entry[http://users.telenet.be/it3.consultants.bvba/handouts/ISO9960.html] (one version with big-endian integers, one version with little-endian integers[http://en.wikipedia.org/wiki/Endianness]) which leads to many kilobytes of wasted space.<br />
** Its file tables are fixed-size, and there are therefore limitations on how many folders you can have, what names you can give your files, how big your files can be etc. For instance, files can only have names that are max 31 characters long, all upper-case, and limited to the ASCII character encoding, and an ISO does only support a folder depth of 8[http://www.on-time.com/rtos-32-docs/rtfiles-32/programming-manual/iso-9660/]. To support more, the Joliet file system extension is needed (or isofs won't recognize all file paths), and the Joliet header can be placed practically anywhere in the file, which means additional seek times.<br />
** It needs various file system extensions to behave correctly, like for example [http://en.wikipedia.org/wiki/Rock_Ridge Rock Ridge Interchange Protocol] and [http://en.wikipedia.org/wiki/Joliet_(file_system) Joliet]. Without them, it becomes pretty useless as demonstrated above, and with them, it becomes difficult to read by a tool (You can only use isofs! There are no other tools out there for programmers to use!).<br />
** It's very difficult to use since you need special ISO making tools to create the image. ISO is a relatively rare format if you're not technically inclined.<br />
** If you want to add or remove files from it, it's near impossible if you use a compact ISO file, since there's no way that you can "expand" an existing ISO file.<br />
* The PND header data is at the end (or in the middle of the file if a screenshot is included) which makes it impossible for e.g. libmagic to recognize the PND file. It will instead recognize it as an ISO file. Having the header data at the end also makes it take a very long time to find the data, making tools and the libpnd library very inefficient.<br />
* The PND file uses a custom XML format for its metadata. There's no reason to do this, especially since the established ".desktop" file format fills exactly the same function.<br />
(note: ".desktop" does not contain all metadata we have wanted over time, but it's of course possible to add extensions à la "X-Pandora-Whatever=xyz")<br />
* There's no "index" for the PND file. The whole file has to be scanned (albeit backwards) to find a PXML file, and there's a big chance for false positives etc.<br />
* Data is just appended linearly to the file so there's no order. If the format is to be extended (to e.g. include an icon file after the screenshot file), should the data just be appended as well?<br />
* UTF-8 is strictly the only encoding that is supported. If you make your PXML on a Windows machine, it won't work.<br />
<br />
== Proposed revisions ==<br />
=== Step 1: File system ===<br />
The file system for PNDs should be replaced by the ''uncompressed'' ZIP archive format. ZIP has the advantage that it's incredibly compact, and uncompressed ZIP makes it possible to read data from the file without having to do any decompression.<br />
<br />
ZIP files are mountable using various implementations of zipfs, and most of these implementations won't store files in memory when they are being read if the ZIP file is uncompressed.<br />
<br />
=== Step 2: Metadata ===<br />
PND files should no longer require special tools for them to be created. Therefore, since ZIP files support random access on files, it shouldn't be necessary to append/prepend metadata to the file. The user simply includes the "PXML.xml" and "preview.png" files inside of the ZIP, and any tools that need information about the PND can simply go to the central directory of the ZIP file (or use a simple ZIP library to do it for them) and get the location of the file inside of the ZIP. This should also dramatically decrease loading times for PNDs.<br />
<br />
Note; performance testing is needed; at the time we started, zip was enormously slower than plain ISO; with driver changes, it may or may not be so, hence our adopted multiple-filesystem-type system. zipfs is one possible option.<br />
<br />
=== Step 3: Metadata format ===<br />
Current PND files include so-called "PXML.xml" files. These files have a custom XML format that has a strange structure.<br />
<br />
These files should be replaced by ".desktop" files. A PND can contain one or more ".desktop" files in its root directory that specify how an application should be launched. PND tools simply use all ".desktop" files they can find in the PND when creating launchers for the contents of the PND.<br />
<br />
== Advantages ==<br />
* There don't have to be ''any'' special tools for reading PND files. The package can be run on any platform using any programming language that can read ZIP files.<br />
* We can use existing facilities to manage the launching of applications. The ".desktop" files can basically be copied without modification into standard locations of the system, and all launchers will become aware of them.<br />
* Reading PND files becomes easier (because of better tool support... sorry, but you can't link to libpnd in all programming languages) and quicker (becuase of the central directory for random file access).<br />
<br />
=== Benchmarks ===<br />
A benchmark was carried through to measure the performance of various package implementations.<br />
The compared systems were:<br />
* ZIP-based uncompressed packages using fuse-zip (a zipfs implementation) to read the files<br />
* ZIP-based compressed packages also using fuse-zip.<br />
* ISO8859-based packages using isofs<br />
* CramFS-based packages.<br />
<br />
A total of 9 files were generated for the test. They have sizes 1, 2, 3, 4, 16, 32, 64, 128 and 256 MB, and were generated via "dd if=/dev/urandom of=$file bs=1M count=$size".<br />
<br />
Because the files contain random data, they did not compress very well. The resulting files were:<br />
* "zipcompressed.zip" 511.1 MB<br />
* "zipuncompressed.zip" 511.0 MB<br />
* "iso.iso" 511.3 MB<br />
* "cramfs.image" 95.4 MB (WOW!)<br />
<br />
Some notes:<br />
* No write tests were done for obvious reasons<br />
* No random access tests were done since the Pandora will use SD cards and thus not get penalized from random access, and that's not what we're interested in anyways.<br />
* Before each test, the commands "sync; echo 3 > /proc/sys/vm/drop_caches" were run.<br />
<br />
==== First test: linear read time ====<br />
Command: "time cat mountpoint/* > /dev/null"<br />
<br />
* ISO: 8.119 seconds<br />
* CramFS: 1.488 seconds (WOW!)<br />
* Zip (compressed): 8.535 seconds<br />
* Zip (uncompressed): 8.290 seconds<br />
<br />
==== Second test: RAM usage ====<br />
-- TODO, have to find a good method of measuring this --<br />
<br />
== Remaining issues ==<br />
* How should libpnd tell the difference between old and new PND files? Should it depend on the "file" tool, should it run its own recognition algorithm, or what? Should the extension be changed to e.g. ".box" (Which was proposed in a thread)?<br />
* How to make sure that the ZIP files are uncompressed? Should we provide a script that "uncompresses" an ordinary ZIP file?<br />
* Is cramfs better than ZIP?<br />
** cramfs cannot be uncompressed like ZIP can<br />
** If you compare compressed ZIP and cramfs, cramfs is more efficient.<br />
** cramfs is harder to use than ZIP (for the developer).<br />
** cramfs ''needs'' to have all of its opened files in memory.<br />
<br />
== Upgrade path from the old PND format ==<br />
A simple script can be written that extracts the old PND file, including the screenshot and the PXML.xml file. The PXML.xml file is then converted to one or many .desktop files, and the desktop files, the preview.png file, and the package contents are compacted into a ZIP that then is renamed to "*.pnd"<br />
<br />
== Usage scenario ==<br />
=== Repackaging of an application from another package format ===<br />
* The user grabs the package for the application<br />
* He dumps the executable and all required libraries into a folder<br />
* He dumps a "screenshot.png" file into the folder that he's made<br />
* He copies the ".desktop" file(s) for the application from the old package, opens them, replaces "Exec=/usr/bin/bla" with "Exec=./bla" and saves them in the directory<br />
* He uses a ZIP archiver to make a ZIP out of the folder, making sure that he sets the "uncompressed" option.<br />
<br />
=== Creating PNDs as part of a build process ===<br />
* The build tool creates a directory with all of the necessary information like above and invokes the "zip" utility to compress the folder.<br />
<br />
=== Accessing a PND's contents for the... ===<br />
<br />
==== ...lazy programmer ====<br />
* The programmer extracts the PND into a folder via libzip and accesses its contents.<br />
<br />
==== ...pragmatic programmer ====<br />
* The programmer uses zipfs to mount the ZIP and accesses the file's contents.<br />
<br />
==== ...smart performance-aware programmer ====<br />
* The programmer fseek()'s the ZIP file until he finds 0x04034b50.<br />
* He jumps 18 bytes forward and reads the int at that location, storing it in "length".<br />
* He jumps 4 bytes and checks if this int matches "length". If it doesn't, it means that the file is compressed, and an error is reported.<br />
* He then jumps forward 4 bytes and stores the short at that location in a variable "nameLength".<br />
* Another 2-byte jump gets the short "extensionsLength".<br />
* He then jumps 2 bytes and reads "nameLength" amount of bytes from the file.<br />
* He then uses strcmp() to see if this string matches the sought-after file. If not, he continues the fseek().<br />
* The programmer now skips "entensionsLength" amount of bytes.<br />
* The programmer reads "length" amount of bytes from the file, and uses this as the sought-after file data.<br />
<br />
==== ...performance fascist/programmer who wants low seek times ====<br />
* The programmer fseek()'s the file from the end until he finds 0x02014b50.<br />
* He then uses the following table to get the information he needs:<br />
{|class="wikitable"<br />
|+ ZIP central directory file header<br />
|-<br />
! Offset !! Bytes !! Description<br />
|-<br />
| 0 || 4 || Central directory file header signature = 0x02014b50<br />
|-<br />
| 4 || 2 || Version made by<br />
|-<br />
| 6 || 2 || Version needed to extract (minimum)<br />
|-<br />
| 8 || 2 || General purpose bit flag<br />
|-<br />
| 10 || 2 || Compression method<br />
|-<br />
| 12 || 2 || File last modification time<br />
|-<br />
| 14 || 2 || File last modification date<br />
|-<br />
| 16 || 4 || CRC-32<br />
|-<br />
| 20 || 4 || Compressed size<br />
|-<br />
| 24 || 4 || Uncompressed size<br />
|-<br />
| 28 || 2 || File name length (''n'')<br />
|-<br />
| 30 || 2 || Extra field length (''m'')<br />
|-<br />
| 32 || 2 || File comment length (''k'')<br />
|-<br />
| 34 || 2 || Disk number where file starts<br />
|-<br />
| 36 || 2 || Internal file attributes<br />
|-<br />
| 38 || 4 || External file attributes<br />
|-<br />
| 42 || 4 || Relative offset of local file header<br />
|-<br />
| 46 || ''n'' || File name<br />
|-<br />
| 46+''n'' || ''m'' || Extra field<br />
|-<br />
| 46+''n''+''m'' || ''k'' || File comment<br />
|}<br />
<br />
* The relative file offset can then be used to jump to the file in question. The file is then scanned as described in the previous section.<br />
<br />
== The 2014/Pyra effort ==<br />
<br />
A new format is being developed for the DragonBox Pyra handheld. Here are the requirements/notes:<br />
<br />
The compatibility layer should have its own table?<br />
<br />
{| class="wikitable"<br />
|-<br />
! Requirement !! Solutions !! Importance !! Status (decided, undecided, post-proned, won't fix, ...) !! Notes<br />
|-<br />
| Should the new format be supported by Pandora AND Pyra? ||<br />
* Yes<br />
* no<br />
|| Very high || ||<br />
* http://boards.openpandora.org/topic/15530-the-software-side/page-7#entry310691<br />
* This decision affects several of the requirements and possible solutions on this table and should be decided sooner<br />
|-<br />
| API/Daemon to switch between joystick/keyboard mode || || || || Not exactly a PND-error<br />
|-<br />
| Easier to parse than current PNDs and less fragile ||<br />
* UBSON header and metadata + icon + screenshots + cramfs<br />
* tar filesystem<br />
** more future proof<br />
** easier for shell scripts<br />
*** with dd you can extract the tar part and so<br />
** filenames and file permissions are overhead<br />
*** But the overhead is too small to be significant?<br />
** easier to modify after-compiled<br />
*** not really a problem, because the user should be contributing to the recipe anyway<br />
*** a specialized to can be create if another solution is chosen<br />
* zipfs uncompressed filesystem<br />
** See above sections<br />
|| || ||<br />
|-<br />
| Should the metadata include packages dependencies? ||<br />
* Yes<br />
** apt-get install package --print-uris will print download uris for dep packages. Using this uris download packages and extract all (dpkg -x 1.deb ./extracted)<br />
*** Not enough. It'll need two passes between an online and an offline system. It's impossible to be sure that an offline installation will work in one pass without including the whole repo: http://boards.openpandora.org/topic/15530-the-software-side/page-9#entry311602<br />
** EvilDragon voted for yes-and-no? http://boards.openpandora.org/topic/15530-the-software-side/page-5#entry310590<br />
*** Or does it mean a minimum "os-version required" metadata on the package and let the user update the system to have the latest libraries available?<br />
*** Possibly he doesn't care and just want something that just works: http://boards.openpandora.org/topic/15574-confliting-requirements/page-2#entry311580<br />
** http://boards.openpandora.org/topic/15530-the-software-side/page-5#entry310627<br />
* No.<br />
** EvilDragon commented that he wanna a plug-and-play-and-works system ( http://boards.openpandora.org/topic/15530-the-software-side/page-3#entry310412 )<br />
** If the user is lacking of internet access, "plug-and-play" package won't work<br />
** .deb for general-purpose software<br />
* set of dependencies that system is required to have (SDL, ...)<br />
* Complete user-space like containment features (see http://www.docker.io/ for an example)<br />
|| || Undecided ||<br />
|-<br />
| It should be click-and-run ||<br />
* binfmt<br />
* Sheebang<br />
* File (extension) association<br />
|| || ||<br />
|-<br />
| Less duplication of .desktop and PXML files || || || ||<br />
|-<br />
| AUR-like system:<br />
* Building recipes is easier to replicate and contribute<br />
* If maintainer is jerk, moderator can move his maintanership<br />
|| Fork makepkg || || [http://boards.openpandora.org/topic/15530-the-software-side/page-7#entry310691 Cloudef is working on this task] ||<br />
|-<br />
| Guideline for consistent use of gaming buttons:<br />
* It should reduce the need of patches on top of the ported software<br />
||<br />
* A keyboard/joystick remapping function<br />
* A guideline for software developers<br />
|| || || No single-solution, but a set of them<br />
|-<br />
| A compatibility layer for the current *.PND system:<br />
* The more transparent, the better<br />
||<br />
* Dual-boot<br />
* Virtual Machine<br />
* Tune Debian system to include all libraries used in current SuperZaxxon<br />
** OpenEmbedded it's based on the Debian, afterall<br />
** Probably won't happen because of hard/soft float point library issues<br />
* chroot<br />
** chroot doesn't start services like the real system, then the services of the chroot must be replaced/configured<br />
|| High || ||<br />
* Pyra will have larger internal space, then duplicate system isn't a large problem<br />
* For current OpenPandora, we can already run Debian on SD-card.<br />
** Maybe we can provide Pyra apps for OpenPandora using the Debian-on-sd-card opposed to Debian-on-internal-space<br />
|-<br />
| Cross-building? || || || Undecided ||<br />
|-<br />
| Filesystem type ||<br />
* Cramfs<br />
* Squashfilesystem<br />
|| || ||<br />
|-<br />
| Home-dir of applications ||<br />
* Maybe on per SD-card<br />
* Union-fs?<br />
* overridable by env var?<br />
|| || ||<br />
|-<br />
| Overridable scripts ||<br />
* By creating files with the right name, you can override files in the package (e.g. scripts)<br />
* Yes:<br />
** Cool for hackers<br />
* No:<br />
** Necessitates unionFS.<br />
** Not absolutely necessary - hackers can just rebuild the package.<br />
|| || ||<br />
|-<br />
| File-extension? ||<br />
* .pyra<br />
** We're not using fucking DOS for lord-sake, what's the point of use 3-letters extension?<br />
* .pyd for pyra-data<br />
* .pyr<br />
* .pp for pyra program<br />
* .pa for pyra app<br />
* .drp for dragonbox pyra<br />
** Why limit this to just the Pyra? Could be useful in future systems/elsewhere too.<br />
* ...<br />
|| || undecided. || Maybe a series of polls on the boards?<br />
|-<br />
| Keep what is good || Document all of the good features and port the content to this table || High || Not started ||<br />
|-<br />
| Extra bonus points for a design that can easily be used on other distros and devices as well || || Optional || Optional || Optional<br />
|-<br />
| Multiple binaries/icons per app || || || || Somebody (lost on the sea of comments) didn't like this design, but it should be possible for special software<br />
|-<br />
| Delta packages || || || || http://boards.openpandora.org/topic/15530-the-software-side/page-9#entry311600<br />
|-<br />
| PATH handling || || || || http://boards.openpandora.org/topic/15530-the-software-side/page-9#entry311618<br />
|}<br />
<br />
[[Category:PND]]<br />
[[Category:Ideas and proposals]]</div>Vinipsmakerhttps://pandorawiki.org/index.php?title=New_PND_format&diff=29472New PND format2014-02-05T21:43:50Z<p>Vinipsmaker: new requirement: path handling</p>
<hr />
<div>The current PND format has some shortcomings as listed below. This page should serve as a discussion page/white board for how the format could be improved.<br />
<br />
''' Proposal for discussion, this is just some opening shots '''<br />
<br />
== Current situation ==<br />
The current (ISO-based) PND format has the following shortcomings:<br />
<br />
* It currently uses ISO, SquashFS, and other filesystems... This means that there's no standard to follow, and libpnd needs to carry around support for a multitude of file systems.<br />
* It (often, but not always) uses the [http://en.wikipedia.org/wiki/ISO_9660 ISO file system] which is inefficient at storing data, because:<br />
** It contains duplicate headers for each file entry[http://users.telenet.be/it3.consultants.bvba/handouts/ISO9960.html] (one version with big-endian integers, one version with little-endian integers[http://en.wikipedia.org/wiki/Endianness]) which leads to many kilobytes of wasted space.<br />
** Its file tables are fixed-size, and there are therefore limitations on how many folders you can have, what names you can give your files, how big your files can be etc. For instance, files can only have names that are max 31 characters long, all upper-case, and limited to the ASCII character encoding, and an ISO does only support a folder depth of 8[http://www.on-time.com/rtos-32-docs/rtfiles-32/programming-manual/iso-9660/]. To support more, the Joliet file system extension is needed (or isofs won't recognize all file paths), and the Joliet header can be placed practically anywhere in the file, which means additional seek times.<br />
** It needs various file system extensions to behave correctly, like for example [http://en.wikipedia.org/wiki/Rock_Ridge Rock Ridge Interchange Protocol] and [http://en.wikipedia.org/wiki/Joliet_(file_system) Joliet]. Without them, it becomes pretty useless as demonstrated above, and with them, it becomes difficult to read by a tool (You can only use isofs! There are no other tools out there for programmers to use!).<br />
** It's very difficult to use since you need special ISO making tools to create the image. ISO is a relatively rare format if you're not technically inclined.<br />
** If you want to add or remove files from it, it's near impossible if you use a compact ISO file, since there's no way that you can "expand" an existing ISO file.<br />
* The PND header data is at the end (or in the middle of the file if a screenshot is included) which makes it impossible for e.g. libmagic to recognize the PND file. It will instead recognize it as an ISO file. Having the header data at the end also makes it take a very long time to find the data, making tools and the libpnd library very inefficient.<br />
* The PND file uses a custom XML format for its metadata. There's no reason to do this, especially since the established ".desktop" file format fills exactly the same function.<br />
(note: ".desktop" does not contain all metadata we have wanted over time, but it's of course possible to add extensions à la "X-Pandora-Whatever=xyz")<br />
* There's no "index" for the PND file. The whole file has to be scanned (albeit backwards) to find a PXML file, and there's a big chance for false positives etc.<br />
* Data is just appended linearly to the file so there's no order. If the format is to be extended (to e.g. include an icon file after the screenshot file), should the data just be appended as well?<br />
* UTF-8 is strictly the only encoding that is supported. If you make your PXML on a Windows machine, it won't work.<br />
<br />
== Proposed revisions ==<br />
=== Step 1: File system ===<br />
The file system for PNDs should be replaced by the ''uncompressed'' ZIP archive format. ZIP has the advantage that it's incredibly compact, and uncompressed ZIP makes it possible to read data from the file without having to do any decompression.<br />
<br />
ZIP files are mountable using various implementations of zipfs, and most of these implementations won't store files in memory when they are being read if the ZIP file is uncompressed.<br />
<br />
=== Step 2: Metadata ===<br />
PND files should no longer require special tools for them to be created. Therefore, since ZIP files support random access on files, it shouldn't be necessary to append/prepend metadata to the file. The user simply includes the "PXML.xml" and "preview.png" files inside of the ZIP, and any tools that need information about the PND can simply go to the central directory of the ZIP file (or use a simple ZIP library to do it for them) and get the location of the file inside of the ZIP. This should also dramatically decrease loading times for PNDs.<br />
<br />
Note; performance testing is needed; at the time we started, zip was enormously slower than plain ISO; with driver changes, it may or may not be so, hence our adopted multiple-filesystem-type system. zipfs is one possible option.<br />
<br />
=== Step 3: Metadata format ===<br />
Current PND files include so-called "PXML.xml" files. These files have a custom XML format that has a strange structure.<br />
<br />
These files should be replaced by ".desktop" files. A PND can contain one or more ".desktop" files in its root directory that specify how an application should be launched. PND tools simply use all ".desktop" files they can find in the PND when creating launchers for the contents of the PND.<br />
<br />
== Advantages ==<br />
* There don't have to be ''any'' special tools for reading PND files. The package can be run on any platform using any programming language that can read ZIP files.<br />
* We can use existing facilities to manage the launching of applications. The ".desktop" files can basically be copied without modification into standard locations of the system, and all launchers will become aware of them.<br />
* Reading PND files becomes easier (because of better tool support... sorry, but you can't link to libpnd in all programming languages) and quicker (becuase of the central directory for random file access).<br />
<br />
=== Benchmarks ===<br />
A benchmark was carried through to measure the performance of various package implementations.<br />
The compared systems were:<br />
* ZIP-based uncompressed packages using fuse-zip (a zipfs implementation) to read the files<br />
* ZIP-based compressed packages also using fuse-zip.<br />
* ISO8859-based packages using isofs<br />
* CramFS-based packages.<br />
<br />
A total of 9 files were generated for the test. They have sizes 1, 2, 3, 4, 16, 32, 64, 128 and 256 MB, and were generated via "dd if=/dev/urandom of=$file bs=1M count=$size".<br />
<br />
Because the files contain random data, they did not compress very well. The resulting files were:<br />
* "zipcompressed.zip" 511.1 MB<br />
* "zipuncompressed.zip" 511.0 MB<br />
* "iso.iso" 511.3 MB<br />
* "cramfs.image" 95.4 MB (WOW!)<br />
<br />
Some notes:<br />
* No write tests were done for obvious reasons<br />
* No random access tests were done since the Pandora will use SD cards and thus not get penalized from random access, and that's not what we're interested in anyways.<br />
* Before each test, the commands "sync; echo 3 > /proc/sys/vm/drop_caches" were run.<br />
<br />
==== First test: linear read time ====<br />
Command: "time cat mountpoint/* > /dev/null"<br />
<br />
* ISO: 8.119 seconds<br />
* CramFS: 1.488 seconds (WOW!)<br />
* Zip (compressed): 8.535 seconds<br />
* Zip (uncompressed): 8.290 seconds<br />
<br />
==== Second test: RAM usage ====<br />
-- TODO, have to find a good method of measuring this --<br />
<br />
== Remaining issues ==<br />
* How should libpnd tell the difference between old and new PND files? Should it depend on the "file" tool, should it run its own recognition algorithm, or what? Should the extension be changed to e.g. ".box" (Which was proposed in a thread)?<br />
* How to make sure that the ZIP files are uncompressed? Should we provide a script that "uncompresses" an ordinary ZIP file?<br />
* Is cramfs better than ZIP?<br />
** cramfs cannot be uncompressed like ZIP can<br />
** If you compare compressed ZIP and cramfs, cramfs is more efficient.<br />
** cramfs is harder to use than ZIP (for the developer).<br />
** cramfs ''needs'' to have all of its opened files in memory.<br />
<br />
== Upgrade path from the old PND format ==<br />
A simple script can be written that extracts the old PND file, including the screenshot and the PXML.xml file. The PXML.xml file is then converted to one or many .desktop files, and the desktop files, the preview.png file, and the package contents are compacted into a ZIP that then is renamed to "*.pnd"<br />
<br />
== Usage scenario ==<br />
=== Repackaging of an application from another package format ===<br />
* The user grabs the package for the application<br />
* He dumps the executable and all required libraries into a folder<br />
* He dumps a "screenshot.png" file into the folder that he's made<br />
* He copies the ".desktop" file(s) for the application from the old package, opens them, replaces "Exec=/usr/bin/bla" with "Exec=./bla" and saves them in the directory<br />
* He uses a ZIP archiver to make a ZIP out of the folder, making sure that he sets the "uncompressed" option.<br />
<br />
=== Creating PNDs as part of a build process ===<br />
* The build tool creates a directory with all of the necessary information like above and invokes the "zip" utility to compress the folder.<br />
<br />
=== Accessing a PND's contents for the... ===<br />
<br />
==== ...lazy programmer ====<br />
* The programmer extracts the PND into a folder via libzip and accesses its contents.<br />
<br />
==== ...pragmatic programmer ====<br />
* The programmer uses zipfs to mount the ZIP and accesses the file's contents.<br />
<br />
==== ...smart performance-aware programmer ====<br />
* The programmer fseek()'s the ZIP file until he finds 0x04034b50.<br />
* He jumps 18 bytes forward and reads the int at that location, storing it in "length".<br />
* He jumps 4 bytes and checks if this int matches "length". If it doesn't, it means that the file is compressed, and an error is reported.<br />
* He then jumps forward 4 bytes and stores the short at that location in a variable "nameLength".<br />
* Another 2-byte jump gets the short "extensionsLength".<br />
* He then jumps 2 bytes and reads "nameLength" amount of bytes from the file.<br />
* He then uses strcmp() to see if this string matches the sought-after file. If not, he continues the fseek().<br />
* The programmer now skips "entensionsLength" amount of bytes.<br />
* The programmer reads "length" amount of bytes from the file, and uses this as the sought-after file data.<br />
<br />
==== ...performance fascist/programmer who wants low seek times ====<br />
* The programmer fseek()'s the file from the end until he finds 0x02014b50.<br />
* He then uses the following table to get the information he needs:<br />
{|class="wikitable"<br />
|+ ZIP central directory file header<br />
|-<br />
! Offset !! Bytes !! Description<br />
|-<br />
| 0 || 4 || Central directory file header signature = 0x02014b50<br />
|-<br />
| 4 || 2 || Version made by<br />
|-<br />
| 6 || 2 || Version needed to extract (minimum)<br />
|-<br />
| 8 || 2 || General purpose bit flag<br />
|-<br />
| 10 || 2 || Compression method<br />
|-<br />
| 12 || 2 || File last modification time<br />
|-<br />
| 14 || 2 || File last modification date<br />
|-<br />
| 16 || 4 || CRC-32<br />
|-<br />
| 20 || 4 || Compressed size<br />
|-<br />
| 24 || 4 || Uncompressed size<br />
|-<br />
| 28 || 2 || File name length (''n'')<br />
|-<br />
| 30 || 2 || Extra field length (''m'')<br />
|-<br />
| 32 || 2 || File comment length (''k'')<br />
|-<br />
| 34 || 2 || Disk number where file starts<br />
|-<br />
| 36 || 2 || Internal file attributes<br />
|-<br />
| 38 || 4 || External file attributes<br />
|-<br />
| 42 || 4 || Relative offset of local file header<br />
|-<br />
| 46 || ''n'' || File name<br />
|-<br />
| 46+''n'' || ''m'' || Extra field<br />
|-<br />
| 46+''n''+''m'' || ''k'' || File comment<br />
|}<br />
<br />
* The relative file offset can then be used to jump to the file in question. The file is then scanned as described in the previous section.<br />
<br />
== The 2014/Pyra effort ==<br />
<br />
A new format is being developed for the DragonBox Pyra handheld. Here are the requirements/notes:<br />
<br />
The compatibility layer should have its own table?<br />
<br />
{| class="wikitable"<br />
|-<br />
! Requirement !! Solutions !! Importance !! Status (decided, undecided, post-proned, won't fix, ...) !! Notes<br />
|-<br />
| Should the new format be supported by Pandora AND Pyra? ||<br />
* Yes<br />
* no<br />
|| Very high || ||<br />
* http://boards.openpandora.org/topic/15530-the-software-side/page-7#entry310691<br />
* This decision affects several of the requirements and possible solutions on this table and should be decided sooner<br />
|-<br />
| API/Daemon to switch between joystick/keyboard mode || || || || Not exactly a PND-error<br />
|-<br />
| Easier to parse than current PNDs and less fragile ||<br />
* UBSON header and metadata + icon + screenshots + cramfs<br />
* tar filesystem<br />
** more future proof<br />
** easier for shell scripts<br />
*** with dd you can extract the tar part and so<br />
** filenames and file permissions are overhead<br />
*** But the overhead is too small to be significant?<br />
** easier to modify after-compiled<br />
*** not really a problem, because the user should be contributing to the recipe anyway<br />
*** a specialized to can be create if another solution is chosen<br />
* zipfs uncompressed filesystem<br />
** See above sections<br />
|| || ||<br />
|-<br />
| Should the metadata include packages dependencies? ||<br />
* Yes<br />
** apt-get install package --print-uris will print download uris for dep packages. Using this uris download packages and extract all (dpkg -x 1.deb ./extracted)<br />
*** Not enough. It'll need two passes between an online and an offline system. It's impossible to be sure that an offline installation will work in one pass without including the whole repo: http://boards.openpandora.org/topic/15530-the-software-side/page-9#entry311602<br />
** EvilDragon voted for yes-and-no? http://boards.openpandora.org/topic/15530-the-software-side/page-5#entry310590<br />
*** Or does it mean a minimum "os-version required" metadata on the package and let the user update the system to have the latest libraries available?<br />
** http://boards.openpandora.org/topic/15530-the-software-side/page-5#entry310627<br />
* No.<br />
** EvilDragon commented that he wanna a plug-and-play-and-works system ( http://boards.openpandora.org/topic/15530-the-software-side/page-3#entry310412 )<br />
** If the user is lacking of internet access, "plug-and-play" package won't work<br />
** .deb for general-purpose software<br />
* set of dependencies that system is required to have (SDL, ...)<br />
* Complete user-space like containment features (see http://www.docker.io/ for an example)<br />
|| || Undecided ||<br />
|-<br />
| It should be click-and-run ||<br />
* binfmt<br />
* Sheebang<br />
* File (extension) association<br />
|| || ||<br />
|-<br />
| Less duplication of .desktop and PXML files || || || ||<br />
|-<br />
| AUR-like system:<br />
* Building recipes is easier to replicate and contribute<br />
* If maintainer is jerk, moderator can move his maintanership<br />
|| Fork makepkg || || [http://boards.openpandora.org/topic/15530-the-software-side/page-7#entry310691 Cloudef is working on this task] ||<br />
|-<br />
| Guideline for consistent use of gaming buttons:<br />
* It should reduce the need of patches on top of the ported software<br />
||<br />
* A keyboard/joystick remapping function<br />
* A guideline for software developers<br />
|| || || No single-solution, but a set of them<br />
|-<br />
| A compatibility layer for the current *.PND system:<br />
* The more transparent, the better<br />
||<br />
* Dual-boot<br />
* Virtual Machine<br />
* Tune Debian system to include all libraries used in current SuperZaxxon<br />
** OpenEmbedded it's based on the Debian, afterall<br />
** Probably won't happen because of hard/soft float point library issues<br />
* chroot<br />
** chroot doesn't start services like the real system, then the services of the chroot must be replaced/configured<br />
|| High || ||<br />
* Pyra will have larger internal space, then duplicate system isn't a large problem<br />
* For current OpenPandora, we can already run Debian on SD-card.<br />
** Maybe we can provide Pyra apps for OpenPandora using the Debian-on-sd-card opposed to Debian-on-internal-space<br />
|-<br />
| Cross-building? || || || Undecided ||<br />
|-<br />
| Filesystem type ||<br />
* Cramfs<br />
* Squashfilesystem<br />
|| || ||<br />
|-<br />
| Home-dir of applications ||<br />
* Maybe on per SD-card<br />
* Union-fs?<br />
* overridable by env var?<br />
|| || ||<br />
|-<br />
| Overridable scripts ||<br />
* By creating files with the right name, you can override files in the package (e.g. scripts)<br />
* Yes:<br />
** Cool for hackers<br />
* No:<br />
** Necessitates unionFS.<br />
** Not absolutely necessary - hackers can just rebuild the package.<br />
|| || ||<br />
|-<br />
| File-extension? ||<br />
* .pyra<br />
** We're not using fucking DOS for lord-sake, what's the point of use 3-letters extension?<br />
* .pyd for pyra-data<br />
* .pyr<br />
* .pp for pyra program<br />
* .pa for pyra app<br />
* .drp for dragonbox pyra<br />
** Why limit this to just the Pyra? Could be useful in future systems/elsewhere too.<br />
* ...<br />
|| || undecided. || Maybe a series of polls on the boards?<br />
|-<br />
| Keep what is good || Document all of the good features and port the content to this table || High || Not started ||<br />
|-<br />
| Extra bonus points for a design that can easily be used on other distros and devices as well || || Optional || Optional || Optional<br />
|-<br />
| Multiple binaries/icons per app || || || || Somebody (lost on the sea of comments) didn't like this design, but it should be possible for special software<br />
|-<br />
| Delta packages || || || || http://boards.openpandora.org/topic/15530-the-software-side/page-9#entry311600<br />
|-<br />
| PATH handling || || || || http://boards.openpandora.org/topic/15530-the-software-side/page-9#entry311618<br />
|}<br />
<br />
[[Category:PND]]<br />
[[Category:Ideas and proposals]]</div>Vinipsmakerhttps://pandorawiki.org/index.php?title=New_PND_format&diff=29471New PND format2014-02-05T21:38:19Z<p>Vinipsmaker: Correcting information about deps</p>
<hr />
<div>The current PND format has some shortcomings as listed below. This page should serve as a discussion page/white board for how the format could be improved.<br />
<br />
''' Proposal for discussion, this is just some opening shots '''<br />
<br />
== Current situation ==<br />
The current (ISO-based) PND format has the following shortcomings:<br />
<br />
* It currently uses ISO, SquashFS, and other filesystems... This means that there's no standard to follow, and libpnd needs to carry around support for a multitude of file systems.<br />
* It (often, but not always) uses the [http://en.wikipedia.org/wiki/ISO_9660 ISO file system] which is inefficient at storing data, because:<br />
** It contains duplicate headers for each file entry[http://users.telenet.be/it3.consultants.bvba/handouts/ISO9960.html] (one version with big-endian integers, one version with little-endian integers[http://en.wikipedia.org/wiki/Endianness]) which leads to many kilobytes of wasted space.<br />
** Its file tables are fixed-size, and there are therefore limitations on how many folders you can have, what names you can give your files, how big your files can be etc. For instance, files can only have names that are max 31 characters long, all upper-case, and limited to the ASCII character encoding, and an ISO does only support a folder depth of 8[http://www.on-time.com/rtos-32-docs/rtfiles-32/programming-manual/iso-9660/]. To support more, the Joliet file system extension is needed (or isofs won't recognize all file paths), and the Joliet header can be placed practically anywhere in the file, which means additional seek times.<br />
** It needs various file system extensions to behave correctly, like for example [http://en.wikipedia.org/wiki/Rock_Ridge Rock Ridge Interchange Protocol] and [http://en.wikipedia.org/wiki/Joliet_(file_system) Joliet]. Without them, it becomes pretty useless as demonstrated above, and with them, it becomes difficult to read by a tool (You can only use isofs! There are no other tools out there for programmers to use!).<br />
** It's very difficult to use since you need special ISO making tools to create the image. ISO is a relatively rare format if you're not technically inclined.<br />
** If you want to add or remove files from it, it's near impossible if you use a compact ISO file, since there's no way that you can "expand" an existing ISO file.<br />
* The PND header data is at the end (or in the middle of the file if a screenshot is included) which makes it impossible for e.g. libmagic to recognize the PND file. It will instead recognize it as an ISO file. Having the header data at the end also makes it take a very long time to find the data, making tools and the libpnd library very inefficient.<br />
* The PND file uses a custom XML format for its metadata. There's no reason to do this, especially since the established ".desktop" file format fills exactly the same function.<br />
(note: ".desktop" does not contain all metadata we have wanted over time, but it's of course possible to add extensions à la "X-Pandora-Whatever=xyz")<br />
* There's no "index" for the PND file. The whole file has to be scanned (albeit backwards) to find a PXML file, and there's a big chance for false positives etc.<br />
* Data is just appended linearly to the file so there's no order. If the format is to be extended (to e.g. include an icon file after the screenshot file), should the data just be appended as well?<br />
* UTF-8 is strictly the only encoding that is supported. If you make your PXML on a Windows machine, it won't work.<br />
<br />
== Proposed revisions ==<br />
=== Step 1: File system ===<br />
The file system for PNDs should be replaced by the ''uncompressed'' ZIP archive format. ZIP has the advantage that it's incredibly compact, and uncompressed ZIP makes it possible to read data from the file without having to do any decompression.<br />
<br />
ZIP files are mountable using various implementations of zipfs, and most of these implementations won't store files in memory when they are being read if the ZIP file is uncompressed.<br />
<br />
=== Step 2: Metadata ===<br />
PND files should no longer require special tools for them to be created. Therefore, since ZIP files support random access on files, it shouldn't be necessary to append/prepend metadata to the file. The user simply includes the "PXML.xml" and "preview.png" files inside of the ZIP, and any tools that need information about the PND can simply go to the central directory of the ZIP file (or use a simple ZIP library to do it for them) and get the location of the file inside of the ZIP. This should also dramatically decrease loading times for PNDs.<br />
<br />
Note; performance testing is needed; at the time we started, zip was enormously slower than plain ISO; with driver changes, it may or may not be so, hence our adopted multiple-filesystem-type system. zipfs is one possible option.<br />
<br />
=== Step 3: Metadata format ===<br />
Current PND files include so-called "PXML.xml" files. These files have a custom XML format that has a strange structure.<br />
<br />
These files should be replaced by ".desktop" files. A PND can contain one or more ".desktop" files in its root directory that specify how an application should be launched. PND tools simply use all ".desktop" files they can find in the PND when creating launchers for the contents of the PND.<br />
<br />
== Advantages ==<br />
* There don't have to be ''any'' special tools for reading PND files. The package can be run on any platform using any programming language that can read ZIP files.<br />
* We can use existing facilities to manage the launching of applications. The ".desktop" files can basically be copied without modification into standard locations of the system, and all launchers will become aware of them.<br />
* Reading PND files becomes easier (because of better tool support... sorry, but you can't link to libpnd in all programming languages) and quicker (becuase of the central directory for random file access).<br />
<br />
=== Benchmarks ===<br />
A benchmark was carried through to measure the performance of various package implementations.<br />
The compared systems were:<br />
* ZIP-based uncompressed packages using fuse-zip (a zipfs implementation) to read the files<br />
* ZIP-based compressed packages also using fuse-zip.<br />
* ISO8859-based packages using isofs<br />
* CramFS-based packages.<br />
<br />
A total of 9 files were generated for the test. They have sizes 1, 2, 3, 4, 16, 32, 64, 128 and 256 MB, and were generated via "dd if=/dev/urandom of=$file bs=1M count=$size".<br />
<br />
Because the files contain random data, they did not compress very well. The resulting files were:<br />
* "zipcompressed.zip" 511.1 MB<br />
* "zipuncompressed.zip" 511.0 MB<br />
* "iso.iso" 511.3 MB<br />
* "cramfs.image" 95.4 MB (WOW!)<br />
<br />
Some notes:<br />
* No write tests were done for obvious reasons<br />
* No random access tests were done since the Pandora will use SD cards and thus not get penalized from random access, and that's not what we're interested in anyways.<br />
* Before each test, the commands "sync; echo 3 > /proc/sys/vm/drop_caches" were run.<br />
<br />
==== First test: linear read time ====<br />
Command: "time cat mountpoint/* > /dev/null"<br />
<br />
* ISO: 8.119 seconds<br />
* CramFS: 1.488 seconds (WOW!)<br />
* Zip (compressed): 8.535 seconds<br />
* Zip (uncompressed): 8.290 seconds<br />
<br />
==== Second test: RAM usage ====<br />
-- TODO, have to find a good method of measuring this --<br />
<br />
== Remaining issues ==<br />
* How should libpnd tell the difference between old and new PND files? Should it depend on the "file" tool, should it run its own recognition algorithm, or what? Should the extension be changed to e.g. ".box" (Which was proposed in a thread)?<br />
* How to make sure that the ZIP files are uncompressed? Should we provide a script that "uncompresses" an ordinary ZIP file?<br />
* Is cramfs better than ZIP?<br />
** cramfs cannot be uncompressed like ZIP can<br />
** If you compare compressed ZIP and cramfs, cramfs is more efficient.<br />
** cramfs is harder to use than ZIP (for the developer).<br />
** cramfs ''needs'' to have all of its opened files in memory.<br />
<br />
== Upgrade path from the old PND format ==<br />
A simple script can be written that extracts the old PND file, including the screenshot and the PXML.xml file. The PXML.xml file is then converted to one or many .desktop files, and the desktop files, the preview.png file, and the package contents are compacted into a ZIP that then is renamed to "*.pnd"<br />
<br />
== Usage scenario ==<br />
=== Repackaging of an application from another package format ===<br />
* The user grabs the package for the application<br />
* He dumps the executable and all required libraries into a folder<br />
* He dumps a "screenshot.png" file into the folder that he's made<br />
* He copies the ".desktop" file(s) for the application from the old package, opens them, replaces "Exec=/usr/bin/bla" with "Exec=./bla" and saves them in the directory<br />
* He uses a ZIP archiver to make a ZIP out of the folder, making sure that he sets the "uncompressed" option.<br />
<br />
=== Creating PNDs as part of a build process ===<br />
* The build tool creates a directory with all of the necessary information like above and invokes the "zip" utility to compress the folder.<br />
<br />
=== Accessing a PND's contents for the... ===<br />
<br />
==== ...lazy programmer ====<br />
* The programmer extracts the PND into a folder via libzip and accesses its contents.<br />
<br />
==== ...pragmatic programmer ====<br />
* The programmer uses zipfs to mount the ZIP and accesses the file's contents.<br />
<br />
==== ...smart performance-aware programmer ====<br />
* The programmer fseek()'s the ZIP file until he finds 0x04034b50.<br />
* He jumps 18 bytes forward and reads the int at that location, storing it in "length".<br />
* He jumps 4 bytes and checks if this int matches "length". If it doesn't, it means that the file is compressed, and an error is reported.<br />
* He then jumps forward 4 bytes and stores the short at that location in a variable "nameLength".<br />
* Another 2-byte jump gets the short "extensionsLength".<br />
* He then jumps 2 bytes and reads "nameLength" amount of bytes from the file.<br />
* He then uses strcmp() to see if this string matches the sought-after file. If not, he continues the fseek().<br />
* The programmer now skips "entensionsLength" amount of bytes.<br />
* The programmer reads "length" amount of bytes from the file, and uses this as the sought-after file data.<br />
<br />
==== ...performance fascist/programmer who wants low seek times ====<br />
* The programmer fseek()'s the file from the end until he finds 0x02014b50.<br />
* He then uses the following table to get the information he needs:<br />
{|class="wikitable"<br />
|+ ZIP central directory file header<br />
|-<br />
! Offset !! Bytes !! Description<br />
|-<br />
| 0 || 4 || Central directory file header signature = 0x02014b50<br />
|-<br />
| 4 || 2 || Version made by<br />
|-<br />
| 6 || 2 || Version needed to extract (minimum)<br />
|-<br />
| 8 || 2 || General purpose bit flag<br />
|-<br />
| 10 || 2 || Compression method<br />
|-<br />
| 12 || 2 || File last modification time<br />
|-<br />
| 14 || 2 || File last modification date<br />
|-<br />
| 16 || 4 || CRC-32<br />
|-<br />
| 20 || 4 || Compressed size<br />
|-<br />
| 24 || 4 || Uncompressed size<br />
|-<br />
| 28 || 2 || File name length (''n'')<br />
|-<br />
| 30 || 2 || Extra field length (''m'')<br />
|-<br />
| 32 || 2 || File comment length (''k'')<br />
|-<br />
| 34 || 2 || Disk number where file starts<br />
|-<br />
| 36 || 2 || Internal file attributes<br />
|-<br />
| 38 || 4 || External file attributes<br />
|-<br />
| 42 || 4 || Relative offset of local file header<br />
|-<br />
| 46 || ''n'' || File name<br />
|-<br />
| 46+''n'' || ''m'' || Extra field<br />
|-<br />
| 46+''n''+''m'' || ''k'' || File comment<br />
|}<br />
<br />
* The relative file offset can then be used to jump to the file in question. The file is then scanned as described in the previous section.<br />
<br />
== The 2014/Pyra effort ==<br />
<br />
A new format is being developed for the DragonBox Pyra handheld. Here are the requirements/notes:<br />
<br />
The compatibility layer should have its own table?<br />
<br />
{| class="wikitable"<br />
|-<br />
! Requirement !! Solutions !! Importance !! Status (decided, undecided, post-proned, won't fix, ...) !! Notes<br />
|-<br />
| Should the new format be supported by Pandora AND Pyra? ||<br />
* Yes<br />
* no<br />
|| Very high || ||<br />
* http://boards.openpandora.org/topic/15530-the-software-side/page-7#entry310691<br />
* This decision affects several of the requirements and possible solutions on this table and should be decided sooner<br />
|-<br />
| API/Daemon to switch between joystick/keyboard mode || || || || Not exactly a PND-error<br />
|-<br />
| Easier to parse than current PNDs and less fragile ||<br />
* UBSON header and metadata + icon + screenshots + cramfs<br />
* tar filesystem<br />
** more future proof<br />
** easier for shell scripts<br />
*** with dd you can extract the tar part and so<br />
** filenames and file permissions are overhead<br />
*** But the overhead is too small to be significant?<br />
** easier to modify after-compiled<br />
*** not really a problem, because the user should be contributing to the recipe anyway<br />
*** a specialized to can be create if another solution is chosen<br />
* zipfs uncompressed filesystem<br />
** See above sections<br />
|| || ||<br />
|-<br />
| Should the metadata include packages dependencies? ||<br />
* Yes<br />
** apt-get install package --print-uris will print download uris for dep packages. Using this uris download packages and extract all (dpkg -x 1.deb ./extracted)<br />
*** Not enough. It'll need two passes between an online and an offline system. It's impossible to be sure that an offline installation will work in one pass without including the whole repo: http://boards.openpandora.org/topic/15530-the-software-side/page-9#entry311602<br />
** EvilDragon voted for yes-and-no? http://boards.openpandora.org/topic/15530-the-software-side/page-5#entry310590<br />
*** Or does it mean a minimum "os-version required" metadata on the package and let the user update the system to have the latest libraries available?<br />
** http://boards.openpandora.org/topic/15530-the-software-side/page-5#entry310627<br />
* No.<br />
** EvilDragon commented that he wanna a plug-and-play-and-works system ( http://boards.openpandora.org/topic/15530-the-software-side/page-3#entry310412 )<br />
** If the user is lacking of internet access, "plug-and-play" package won't work<br />
** .deb for general-purpose software<br />
* set of dependencies that system is required to have (SDL, ...)<br />
* Complete user-space like containment features (see http://www.docker.io/ for an example)<br />
|| || Undecided ||<br />
|-<br />
| It should be click-and-run ||<br />
* binfmt<br />
* Sheebang<br />
* File (extension) association<br />
|| || ||<br />
|-<br />
| Less duplication of .desktop and PXML files || || || ||<br />
|-<br />
| AUR-like system:<br />
* Building recipes is easier to replicate and contribute<br />
* If maintainer is jerk, moderator can move his maintanership<br />
|| Fork makepkg || || [http://boards.openpandora.org/topic/15530-the-software-side/page-7#entry310691 Cloudef is working on this task] ||<br />
|-<br />
| Guideline for consistent use of gaming buttons:<br />
* It should reduce the need of patches on top of the ported software<br />
||<br />
* A keyboard/joystick remapping function<br />
* A guideline for software developers<br />
|| || || No single-solution, but a set of them<br />
|-<br />
| A compatibility layer for the current *.PND system:<br />
* The more transparent, the better<br />
||<br />
* Dual-boot<br />
* Virtual Machine<br />
* Tune Debian system to include all libraries used in current SuperZaxxon<br />
** OpenEmbedded it's based on the Debian, afterall<br />
** Probably won't happen because of hard/soft float point library issues<br />
* chroot<br />
** chroot doesn't start services like the real system, then the services of the chroot must be replaced/configured<br />
|| High || ||<br />
* Pyra will have larger internal space, then duplicate system isn't a large problem<br />
* For current OpenPandora, we can already run Debian on SD-card.<br />
** Maybe we can provide Pyra apps for OpenPandora using the Debian-on-sd-card opposed to Debian-on-internal-space<br />
|-<br />
| Cross-building? || || || Undecided ||<br />
|-<br />
| Filesystem type ||<br />
* Cramfs<br />
* Squashfilesystem<br />
|| || ||<br />
|-<br />
| Home-dir of applications ||<br />
* Maybe on per SD-card<br />
* Union-fs?<br />
* overridable by env var?<br />
|| || ||<br />
|-<br />
| Overridable scripts ||<br />
* By creating files with the right name, you can override files in the package (e.g. scripts)<br />
* Yes:<br />
** Cool for hackers<br />
* No:<br />
** Necessitates unionFS.<br />
** Not absolutely necessary - hackers can just rebuild the package.<br />
|| || ||<br />
|-<br />
| File-extension? ||<br />
* .pyra<br />
** We're not using fucking DOS for lord-sake, what's the point of use 3-letters extension?<br />
* .pyd for pyra-data<br />
* .pyr<br />
* .pp for pyra program<br />
* .pa for pyra app<br />
* .drp for dragonbox pyra<br />
** Why limit this to just the Pyra? Could be useful in future systems/elsewhere too.<br />
* ...<br />
|| || undecided. || Maybe a series of polls on the boards?<br />
|-<br />
| Keep what is good || Document all of the good features and port the content to this table || High || Not started ||<br />
|-<br />
| Extra bonus points for a design that can easily be used on other distros and devices as well || || Optional || Optional || Optional<br />
|-<br />
| Multiple binaries/icons per app || || || || Somebody (lost on the sea of comments) didn't like this design, but it should be possible for special software<br />
|-<br />
| Delta packages || || || || http://boards.openpandora.org/topic/15530-the-software-side/page-9#entry311600<br />
|}<br />
<br />
[[Category:PND]]<br />
[[Category:Ideas and proposals]]</div>Vinipsmakerhttps://pandorawiki.org/index.php?title=New_PND_format&diff=29470New PND format2014-02-05T21:37:07Z<p>Vinipsmaker: Should include deps?</p>
<hr />
<div>The current PND format has some shortcomings as listed below. This page should serve as a discussion page/white board for how the format could be improved.<br />
<br />
''' Proposal for discussion, this is just some opening shots '''<br />
<br />
== Current situation ==<br />
The current (ISO-based) PND format has the following shortcomings:<br />
<br />
* It currently uses ISO, SquashFS, and other filesystems... This means that there's no standard to follow, and libpnd needs to carry around support for a multitude of file systems.<br />
* It (often, but not always) uses the [http://en.wikipedia.org/wiki/ISO_9660 ISO file system] which is inefficient at storing data, because:<br />
** It contains duplicate headers for each file entry[http://users.telenet.be/it3.consultants.bvba/handouts/ISO9960.html] (one version with big-endian integers, one version with little-endian integers[http://en.wikipedia.org/wiki/Endianness]) which leads to many kilobytes of wasted space.<br />
** Its file tables are fixed-size, and there are therefore limitations on how many folders you can have, what names you can give your files, how big your files can be etc. For instance, files can only have names that are max 31 characters long, all upper-case, and limited to the ASCII character encoding, and an ISO does only support a folder depth of 8[http://www.on-time.com/rtos-32-docs/rtfiles-32/programming-manual/iso-9660/]. To support more, the Joliet file system extension is needed (or isofs won't recognize all file paths), and the Joliet header can be placed practically anywhere in the file, which means additional seek times.<br />
** It needs various file system extensions to behave correctly, like for example [http://en.wikipedia.org/wiki/Rock_Ridge Rock Ridge Interchange Protocol] and [http://en.wikipedia.org/wiki/Joliet_(file_system) Joliet]. Without them, it becomes pretty useless as demonstrated above, and with them, it becomes difficult to read by a tool (You can only use isofs! There are no other tools out there for programmers to use!).<br />
** It's very difficult to use since you need special ISO making tools to create the image. ISO is a relatively rare format if you're not technically inclined.<br />
** If you want to add or remove files from it, it's near impossible if you use a compact ISO file, since there's no way that you can "expand" an existing ISO file.<br />
* The PND header data is at the end (or in the middle of the file if a screenshot is included) which makes it impossible for e.g. libmagic to recognize the PND file. It will instead recognize it as an ISO file. Having the header data at the end also makes it take a very long time to find the data, making tools and the libpnd library very inefficient.<br />
* The PND file uses a custom XML format for its metadata. There's no reason to do this, especially since the established ".desktop" file format fills exactly the same function.<br />
(note: ".desktop" does not contain all metadata we have wanted over time, but it's of course possible to add extensions à la "X-Pandora-Whatever=xyz")<br />
* There's no "index" for the PND file. The whole file has to be scanned (albeit backwards) to find a PXML file, and there's a big chance for false positives etc.<br />
* Data is just appended linearly to the file so there's no order. If the format is to be extended (to e.g. include an icon file after the screenshot file), should the data just be appended as well?<br />
* UTF-8 is strictly the only encoding that is supported. If you make your PXML on a Windows machine, it won't work.<br />
<br />
== Proposed revisions ==<br />
=== Step 1: File system ===<br />
The file system for PNDs should be replaced by the ''uncompressed'' ZIP archive format. ZIP has the advantage that it's incredibly compact, and uncompressed ZIP makes it possible to read data from the file without having to do any decompression.<br />
<br />
ZIP files are mountable using various implementations of zipfs, and most of these implementations won't store files in memory when they are being read if the ZIP file is uncompressed.<br />
<br />
=== Step 2: Metadata ===<br />
PND files should no longer require special tools for them to be created. Therefore, since ZIP files support random access on files, it shouldn't be necessary to append/prepend metadata to the file. The user simply includes the "PXML.xml" and "preview.png" files inside of the ZIP, and any tools that need information about the PND can simply go to the central directory of the ZIP file (or use a simple ZIP library to do it for them) and get the location of the file inside of the ZIP. This should also dramatically decrease loading times for PNDs.<br />
<br />
Note; performance testing is needed; at the time we started, zip was enormously slower than plain ISO; with driver changes, it may or may not be so, hence our adopted multiple-filesystem-type system. zipfs is one possible option.<br />
<br />
=== Step 3: Metadata format ===<br />
Current PND files include so-called "PXML.xml" files. These files have a custom XML format that has a strange structure.<br />
<br />
These files should be replaced by ".desktop" files. A PND can contain one or more ".desktop" files in its root directory that specify how an application should be launched. PND tools simply use all ".desktop" files they can find in the PND when creating launchers for the contents of the PND.<br />
<br />
== Advantages ==<br />
* There don't have to be ''any'' special tools for reading PND files. The package can be run on any platform using any programming language that can read ZIP files.<br />
* We can use existing facilities to manage the launching of applications. The ".desktop" files can basically be copied without modification into standard locations of the system, and all launchers will become aware of them.<br />
* Reading PND files becomes easier (because of better tool support... sorry, but you can't link to libpnd in all programming languages) and quicker (becuase of the central directory for random file access).<br />
<br />
=== Benchmarks ===<br />
A benchmark was carried through to measure the performance of various package implementations.<br />
The compared systems were:<br />
* ZIP-based uncompressed packages using fuse-zip (a zipfs implementation) to read the files<br />
* ZIP-based compressed packages also using fuse-zip.<br />
* ISO8859-based packages using isofs<br />
* CramFS-based packages.<br />
<br />
A total of 9 files were generated for the test. They have sizes 1, 2, 3, 4, 16, 32, 64, 128 and 256 MB, and were generated via "dd if=/dev/urandom of=$file bs=1M count=$size".<br />
<br />
Because the files contain random data, they did not compress very well. The resulting files were:<br />
* "zipcompressed.zip" 511.1 MB<br />
* "zipuncompressed.zip" 511.0 MB<br />
* "iso.iso" 511.3 MB<br />
* "cramfs.image" 95.4 MB (WOW!)<br />
<br />
Some notes:<br />
* No write tests were done for obvious reasons<br />
* No random access tests were done since the Pandora will use SD cards and thus not get penalized from random access, and that's not what we're interested in anyways.<br />
* Before each test, the commands "sync; echo 3 > /proc/sys/vm/drop_caches" were run.<br />
<br />
==== First test: linear read time ====<br />
Command: "time cat mountpoint/* > /dev/null"<br />
<br />
* ISO: 8.119 seconds<br />
* CramFS: 1.488 seconds (WOW!)<br />
* Zip (compressed): 8.535 seconds<br />
* Zip (uncompressed): 8.290 seconds<br />
<br />
==== Second test: RAM usage ====<br />
-- TODO, have to find a good method of measuring this --<br />
<br />
== Remaining issues ==<br />
* How should libpnd tell the difference between old and new PND files? Should it depend on the "file" tool, should it run its own recognition algorithm, or what? Should the extension be changed to e.g. ".box" (Which was proposed in a thread)?<br />
* How to make sure that the ZIP files are uncompressed? Should we provide a script that "uncompresses" an ordinary ZIP file?<br />
* Is cramfs better than ZIP?<br />
** cramfs cannot be uncompressed like ZIP can<br />
** If you compare compressed ZIP and cramfs, cramfs is more efficient.<br />
** cramfs is harder to use than ZIP (for the developer).<br />
** cramfs ''needs'' to have all of its opened files in memory.<br />
<br />
== Upgrade path from the old PND format ==<br />
A simple script can be written that extracts the old PND file, including the screenshot and the PXML.xml file. The PXML.xml file is then converted to one or many .desktop files, and the desktop files, the preview.png file, and the package contents are compacted into a ZIP that then is renamed to "*.pnd"<br />
<br />
== Usage scenario ==<br />
=== Repackaging of an application from another package format ===<br />
* The user grabs the package for the application<br />
* He dumps the executable and all required libraries into a folder<br />
* He dumps a "screenshot.png" file into the folder that he's made<br />
* He copies the ".desktop" file(s) for the application from the old package, opens them, replaces "Exec=/usr/bin/bla" with "Exec=./bla" and saves them in the directory<br />
* He uses a ZIP archiver to make a ZIP out of the folder, making sure that he sets the "uncompressed" option.<br />
<br />
=== Creating PNDs as part of a build process ===<br />
* The build tool creates a directory with all of the necessary information like above and invokes the "zip" utility to compress the folder.<br />
<br />
=== Accessing a PND's contents for the... ===<br />
<br />
==== ...lazy programmer ====<br />
* The programmer extracts the PND into a folder via libzip and accesses its contents.<br />
<br />
==== ...pragmatic programmer ====<br />
* The programmer uses zipfs to mount the ZIP and accesses the file's contents.<br />
<br />
==== ...smart performance-aware programmer ====<br />
* The programmer fseek()'s the ZIP file until he finds 0x04034b50.<br />
* He jumps 18 bytes forward and reads the int at that location, storing it in "length".<br />
* He jumps 4 bytes and checks if this int matches "length". If it doesn't, it means that the file is compressed, and an error is reported.<br />
* He then jumps forward 4 bytes and stores the short at that location in a variable "nameLength".<br />
* Another 2-byte jump gets the short "extensionsLength".<br />
* He then jumps 2 bytes and reads "nameLength" amount of bytes from the file.<br />
* He then uses strcmp() to see if this string matches the sought-after file. If not, he continues the fseek().<br />
* The programmer now skips "entensionsLength" amount of bytes.<br />
* The programmer reads "length" amount of bytes from the file, and uses this as the sought-after file data.<br />
<br />
==== ...performance fascist/programmer who wants low seek times ====<br />
* The programmer fseek()'s the file from the end until he finds 0x02014b50.<br />
* He then uses the following table to get the information he needs:<br />
{|class="wikitable"<br />
|+ ZIP central directory file header<br />
|-<br />
! Offset !! Bytes !! Description<br />
|-<br />
| 0 || 4 || Central directory file header signature = 0x02014b50<br />
|-<br />
| 4 || 2 || Version made by<br />
|-<br />
| 6 || 2 || Version needed to extract (minimum)<br />
|-<br />
| 8 || 2 || General purpose bit flag<br />
|-<br />
| 10 || 2 || Compression method<br />
|-<br />
| 12 || 2 || File last modification time<br />
|-<br />
| 14 || 2 || File last modification date<br />
|-<br />
| 16 || 4 || CRC-32<br />
|-<br />
| 20 || 4 || Compressed size<br />
|-<br />
| 24 || 4 || Uncompressed size<br />
|-<br />
| 28 || 2 || File name length (''n'')<br />
|-<br />
| 30 || 2 || Extra field length (''m'')<br />
|-<br />
| 32 || 2 || File comment length (''k'')<br />
|-<br />
| 34 || 2 || Disk number where file starts<br />
|-<br />
| 36 || 2 || Internal file attributes<br />
|-<br />
| 38 || 4 || External file attributes<br />
|-<br />
| 42 || 4 || Relative offset of local file header<br />
|-<br />
| 46 || ''n'' || File name<br />
|-<br />
| 46+''n'' || ''m'' || Extra field<br />
|-<br />
| 46+''n''+''m'' || ''k'' || File comment<br />
|}<br />
<br />
* The relative file offset can then be used to jump to the file in question. The file is then scanned as described in the previous section.<br />
<br />
== The 2014/Pyra effort ==<br />
<br />
A new format is being developed for the DragonBox Pyra handheld. Here are the requirements/notes:<br />
<br />
The compatibility layer should have its own table?<br />
<br />
{| class="wikitable"<br />
|-<br />
! Requirement !! Solutions !! Importance !! Status (decided, undecided, post-proned, won't fix, ...) !! Notes<br />
|-<br />
| Should the new format be supported by Pandora AND Pyra? ||<br />
* Yes<br />
* no<br />
|| Very high || ||<br />
* http://boards.openpandora.org/topic/15530-the-software-side/page-7#entry310691<br />
* This decision affects several of the requirements and possible solutions on this table and should be decided sooner<br />
|-<br />
| API/Daemon to switch between joystick/keyboard mode || || || || Not exactly a PND-error<br />
|-<br />
| Easier to parse than current PNDs and less fragile ||<br />
* UBSON header and metadata + icon + screenshots + cramfs<br />
* tar filesystem<br />
** more future proof<br />
** easier for shell scripts<br />
*** with dd you can extract the tar part and so<br />
** filenames and file permissions are overhead<br />
*** But the overhead is too small to be significant?<br />
** easier to modify after-compiled<br />
*** not really a problem, because the user should be contributing to the recipe anyway<br />
*** a specialized to can be create if another solution is chosen<br />
* zipfs uncompressed filesystem<br />
** See above sections<br />
|| || ||<br />
|-<br />
| Should the metadata include packages dependencies? ||<br />
* Yes<br />
** apt-get install package --print-uris will print download uris for dep packages. Using this uris download packages and extract all (dpkg -x 1.deb ./extracted)<br />
*** Not enough. It'll need two passes between an online and an offline system. It's impossible to be sure that an offline installation will work without including the whole repo: http://boards.openpandora.org/topic/15530-the-software-side/page-9#entry311602<br />
** EvilDragon voted for yes-and-no? http://boards.openpandora.org/topic/15530-the-software-side/page-5#entry310590<br />
*** Or does it mean a minimum "os-version required" metadata on the package and let the user update the system to have the latest libraries available?<br />
** http://boards.openpandora.org/topic/15530-the-software-side/page-5#entry310627<br />
* No.<br />
** EvilDragon commented that he wanna a plug-and-play-and-works system ( http://boards.openpandora.org/topic/15530-the-software-side/page-3#entry310412 )<br />
** If the user is lacking of internet access, "plug-and-play" package won't work<br />
** .deb for general-purpose software<br />
* set of dependencies that system is required to have (SDL, ...)<br />
* Complete user-space like containment features (see http://www.docker.io/ for an example)<br />
|| || Undecided ||<br />
|-<br />
| It should be click-and-run ||<br />
* binfmt<br />
* Sheebang<br />
* File (extension) association<br />
|| || ||<br />
|-<br />
| Less duplication of .desktop and PXML files || || || ||<br />
|-<br />
| AUR-like system:<br />
* Building recipes is easier to replicate and contribute<br />
* If maintainer is jerk, moderator can move his maintanership<br />
|| Fork makepkg || || [http://boards.openpandora.org/topic/15530-the-software-side/page-7#entry310691 Cloudef is working on this task] ||<br />
|-<br />
| Guideline for consistent use of gaming buttons:<br />
* It should reduce the need of patches on top of the ported software<br />
||<br />
* A keyboard/joystick remapping function<br />
* A guideline for software developers<br />
|| || || No single-solution, but a set of them<br />
|-<br />
| A compatibility layer for the current *.PND system:<br />
* The more transparent, the better<br />
||<br />
* Dual-boot<br />
* Virtual Machine<br />
* Tune Debian system to include all libraries used in current SuperZaxxon<br />
** OpenEmbedded it's based on the Debian, afterall<br />
** Probably won't happen because of hard/soft float point library issues<br />
* chroot<br />
** chroot doesn't start services like the real system, then the services of the chroot must be replaced/configured<br />
|| High || ||<br />
* Pyra will have larger internal space, then duplicate system isn't a large problem<br />
* For current OpenPandora, we can already run Debian on SD-card.<br />
** Maybe we can provide Pyra apps for OpenPandora using the Debian-on-sd-card opposed to Debian-on-internal-space<br />
|-<br />
| Cross-building? || || || Undecided ||<br />
|-<br />
| Filesystem type ||<br />
* Cramfs<br />
* Squashfilesystem<br />
|| || ||<br />
|-<br />
| Home-dir of applications ||<br />
* Maybe on per SD-card<br />
* Union-fs?<br />
* overridable by env var?<br />
|| || ||<br />
|-<br />
| Overridable scripts ||<br />
* By creating files with the right name, you can override files in the package (e.g. scripts)<br />
* Yes:<br />
** Cool for hackers<br />
* No:<br />
** Necessitates unionFS.<br />
** Not absolutely necessary - hackers can just rebuild the package.<br />
|| || ||<br />
|-<br />
| File-extension? ||<br />
* .pyra<br />
** We're not using fucking DOS for lord-sake, what's the point of use 3-letters extension?<br />
* .pyd for pyra-data<br />
* .pyr<br />
* .pp for pyra program<br />
* .pa for pyra app<br />
* .drp for dragonbox pyra<br />
** Why limit this to just the Pyra? Could be useful in future systems/elsewhere too.<br />
* ...<br />
|| || undecided. || Maybe a series of polls on the boards?<br />
|-<br />
| Keep what is good || Document all of the good features and port the content to this table || High || Not started ||<br />
|-<br />
| Extra bonus points for a design that can easily be used on other distros and devices as well || || Optional || Optional || Optional<br />
|-<br />
| Multiple binaries/icons per app || || || || Somebody (lost on the sea of comments) didn't like this design, but it should be possible for special software<br />
|-<br />
| Delta packages || || || || http://boards.openpandora.org/topic/15530-the-software-side/page-9#entry311600<br />
|}<br />
<br />
[[Category:PND]]<br />
[[Category:Ideas and proposals]]</div>Vinipsmakerhttps://pandorawiki.org/index.php?title=New_PND_format&diff=29469New PND format2014-02-05T21:16:03Z<p>Vinipsmaker: Adding delta-packages requirement</p>
<hr />
<div>The current PND format has some shortcomings as listed below. This page should serve as a discussion page/white board for how the format could be improved.<br />
<br />
''' Proposal for discussion, this is just some opening shots '''<br />
<br />
== Current situation ==<br />
The current (ISO-based) PND format has the following shortcomings:<br />
<br />
* It currently uses ISO, SquashFS, and other filesystems... This means that there's no standard to follow, and libpnd needs to carry around support for a multitude of file systems.<br />
* It (often, but not always) uses the [http://en.wikipedia.org/wiki/ISO_9660 ISO file system] which is inefficient at storing data, because:<br />
** It contains duplicate headers for each file entry[http://users.telenet.be/it3.consultants.bvba/handouts/ISO9960.html] (one version with big-endian integers, one version with little-endian integers[http://en.wikipedia.org/wiki/Endianness]) which leads to many kilobytes of wasted space.<br />
** Its file tables are fixed-size, and there are therefore limitations on how many folders you can have, what names you can give your files, how big your files can be etc. For instance, files can only have names that are max 31 characters long, all upper-case, and limited to the ASCII character encoding, and an ISO does only support a folder depth of 8[http://www.on-time.com/rtos-32-docs/rtfiles-32/programming-manual/iso-9660/]. To support more, the Joliet file system extension is needed (or isofs won't recognize all file paths), and the Joliet header can be placed practically anywhere in the file, which means additional seek times.<br />
** It needs various file system extensions to behave correctly, like for example [http://en.wikipedia.org/wiki/Rock_Ridge Rock Ridge Interchange Protocol] and [http://en.wikipedia.org/wiki/Joliet_(file_system) Joliet]. Without them, it becomes pretty useless as demonstrated above, and with them, it becomes difficult to read by a tool (You can only use isofs! There are no other tools out there for programmers to use!).<br />
** It's very difficult to use since you need special ISO making tools to create the image. ISO is a relatively rare format if you're not technically inclined.<br />
** If you want to add or remove files from it, it's near impossible if you use a compact ISO file, since there's no way that you can "expand" an existing ISO file.<br />
* The PND header data is at the end (or in the middle of the file if a screenshot is included) which makes it impossible for e.g. libmagic to recognize the PND file. It will instead recognize it as an ISO file. Having the header data at the end also makes it take a very long time to find the data, making tools and the libpnd library very inefficient.<br />
* The PND file uses a custom XML format for its metadata. There's no reason to do this, especially since the established ".desktop" file format fills exactly the same function.<br />
(note: ".desktop" does not contain all metadata we have wanted over time, but it's of course possible to add extensions à la "X-Pandora-Whatever=xyz")<br />
* There's no "index" for the PND file. The whole file has to be scanned (albeit backwards) to find a PXML file, and there's a big chance for false positives etc.<br />
* Data is just appended linearly to the file so there's no order. If the format is to be extended (to e.g. include an icon file after the screenshot file), should the data just be appended as well?<br />
* UTF-8 is strictly the only encoding that is supported. If you make your PXML on a Windows machine, it won't work.<br />
<br />
== Proposed revisions ==<br />
=== Step 1: File system ===<br />
The file system for PNDs should be replaced by the ''uncompressed'' ZIP archive format. ZIP has the advantage that it's incredibly compact, and uncompressed ZIP makes it possible to read data from the file without having to do any decompression.<br />
<br />
ZIP files are mountable using various implementations of zipfs, and most of these implementations won't store files in memory when they are being read if the ZIP file is uncompressed.<br />
<br />
=== Step 2: Metadata ===<br />
PND files should no longer require special tools for them to be created. Therefore, since ZIP files support random access on files, it shouldn't be necessary to append/prepend metadata to the file. The user simply includes the "PXML.xml" and "preview.png" files inside of the ZIP, and any tools that need information about the PND can simply go to the central directory of the ZIP file (or use a simple ZIP library to do it for them) and get the location of the file inside of the ZIP. This should also dramatically decrease loading times for PNDs.<br />
<br />
Note; performance testing is needed; at the time we started, zip was enormously slower than plain ISO; with driver changes, it may or may not be so, hence our adopted multiple-filesystem-type system. zipfs is one possible option.<br />
<br />
=== Step 3: Metadata format ===<br />
Current PND files include so-called "PXML.xml" files. These files have a custom XML format that has a strange structure.<br />
<br />
These files should be replaced by ".desktop" files. A PND can contain one or more ".desktop" files in its root directory that specify how an application should be launched. PND tools simply use all ".desktop" files they can find in the PND when creating launchers for the contents of the PND.<br />
<br />
== Advantages ==<br />
* There don't have to be ''any'' special tools for reading PND files. The package can be run on any platform using any programming language that can read ZIP files.<br />
* We can use existing facilities to manage the launching of applications. The ".desktop" files can basically be copied without modification into standard locations of the system, and all launchers will become aware of them.<br />
* Reading PND files becomes easier (because of better tool support... sorry, but you can't link to libpnd in all programming languages) and quicker (becuase of the central directory for random file access).<br />
<br />
=== Benchmarks ===<br />
A benchmark was carried through to measure the performance of various package implementations.<br />
The compared systems were:<br />
* ZIP-based uncompressed packages using fuse-zip (a zipfs implementation) to read the files<br />
* ZIP-based compressed packages also using fuse-zip.<br />
* ISO8859-based packages using isofs<br />
* CramFS-based packages.<br />
<br />
A total of 9 files were generated for the test. They have sizes 1, 2, 3, 4, 16, 32, 64, 128 and 256 MB, and were generated via "dd if=/dev/urandom of=$file bs=1M count=$size".<br />
<br />
Because the files contain random data, they did not compress very well. The resulting files were:<br />
* "zipcompressed.zip" 511.1 MB<br />
* "zipuncompressed.zip" 511.0 MB<br />
* "iso.iso" 511.3 MB<br />
* "cramfs.image" 95.4 MB (WOW!)<br />
<br />
Some notes:<br />
* No write tests were done for obvious reasons<br />
* No random access tests were done since the Pandora will use SD cards and thus not get penalized from random access, and that's not what we're interested in anyways.<br />
* Before each test, the commands "sync; echo 3 > /proc/sys/vm/drop_caches" were run.<br />
<br />
==== First test: linear read time ====<br />
Command: "time cat mountpoint/* > /dev/null"<br />
<br />
* ISO: 8.119 seconds<br />
* CramFS: 1.488 seconds (WOW!)<br />
* Zip (compressed): 8.535 seconds<br />
* Zip (uncompressed): 8.290 seconds<br />
<br />
==== Second test: RAM usage ====<br />
-- TODO, have to find a good method of measuring this --<br />
<br />
== Remaining issues ==<br />
* How should libpnd tell the difference between old and new PND files? Should it depend on the "file" tool, should it run its own recognition algorithm, or what? Should the extension be changed to e.g. ".box" (Which was proposed in a thread)?<br />
* How to make sure that the ZIP files are uncompressed? Should we provide a script that "uncompresses" an ordinary ZIP file?<br />
* Is cramfs better than ZIP?<br />
** cramfs cannot be uncompressed like ZIP can<br />
** If you compare compressed ZIP and cramfs, cramfs is more efficient.<br />
** cramfs is harder to use than ZIP (for the developer).<br />
** cramfs ''needs'' to have all of its opened files in memory.<br />
<br />
== Upgrade path from the old PND format ==<br />
A simple script can be written that extracts the old PND file, including the screenshot and the PXML.xml file. The PXML.xml file is then converted to one or many .desktop files, and the desktop files, the preview.png file, and the package contents are compacted into a ZIP that then is renamed to "*.pnd"<br />
<br />
== Usage scenario ==<br />
=== Repackaging of an application from another package format ===<br />
* The user grabs the package for the application<br />
* He dumps the executable and all required libraries into a folder<br />
* He dumps a "screenshot.png" file into the folder that he's made<br />
* He copies the ".desktop" file(s) for the application from the old package, opens them, replaces "Exec=/usr/bin/bla" with "Exec=./bla" and saves them in the directory<br />
* He uses a ZIP archiver to make a ZIP out of the folder, making sure that he sets the "uncompressed" option.<br />
<br />
=== Creating PNDs as part of a build process ===<br />
* The build tool creates a directory with all of the necessary information like above and invokes the "zip" utility to compress the folder.<br />
<br />
=== Accessing a PND's contents for the... ===<br />
<br />
==== ...lazy programmer ====<br />
* The programmer extracts the PND into a folder via libzip and accesses its contents.<br />
<br />
==== ...pragmatic programmer ====<br />
* The programmer uses zipfs to mount the ZIP and accesses the file's contents.<br />
<br />
==== ...smart performance-aware programmer ====<br />
* The programmer fseek()'s the ZIP file until he finds 0x04034b50.<br />
* He jumps 18 bytes forward and reads the int at that location, storing it in "length".<br />
* He jumps 4 bytes and checks if this int matches "length". If it doesn't, it means that the file is compressed, and an error is reported.<br />
* He then jumps forward 4 bytes and stores the short at that location in a variable "nameLength".<br />
* Another 2-byte jump gets the short "extensionsLength".<br />
* He then jumps 2 bytes and reads "nameLength" amount of bytes from the file.<br />
* He then uses strcmp() to see if this string matches the sought-after file. If not, he continues the fseek().<br />
* The programmer now skips "entensionsLength" amount of bytes.<br />
* The programmer reads "length" amount of bytes from the file, and uses this as the sought-after file data.<br />
<br />
==== ...performance fascist/programmer who wants low seek times ====<br />
* The programmer fseek()'s the file from the end until he finds 0x02014b50.<br />
* He then uses the following table to get the information he needs:<br />
{|class="wikitable"<br />
|+ ZIP central directory file header<br />
|-<br />
! Offset !! Bytes !! Description<br />
|-<br />
| 0 || 4 || Central directory file header signature = 0x02014b50<br />
|-<br />
| 4 || 2 || Version made by<br />
|-<br />
| 6 || 2 || Version needed to extract (minimum)<br />
|-<br />
| 8 || 2 || General purpose bit flag<br />
|-<br />
| 10 || 2 || Compression method<br />
|-<br />
| 12 || 2 || File last modification time<br />
|-<br />
| 14 || 2 || File last modification date<br />
|-<br />
| 16 || 4 || CRC-32<br />
|-<br />
| 20 || 4 || Compressed size<br />
|-<br />
| 24 || 4 || Uncompressed size<br />
|-<br />
| 28 || 2 || File name length (''n'')<br />
|-<br />
| 30 || 2 || Extra field length (''m'')<br />
|-<br />
| 32 || 2 || File comment length (''k'')<br />
|-<br />
| 34 || 2 || Disk number where file starts<br />
|-<br />
| 36 || 2 || Internal file attributes<br />
|-<br />
| 38 || 4 || External file attributes<br />
|-<br />
| 42 || 4 || Relative offset of local file header<br />
|-<br />
| 46 || ''n'' || File name<br />
|-<br />
| 46+''n'' || ''m'' || Extra field<br />
|-<br />
| 46+''n''+''m'' || ''k'' || File comment<br />
|}<br />
<br />
* The relative file offset can then be used to jump to the file in question. The file is then scanned as described in the previous section.<br />
<br />
== The 2014/Pyra effort ==<br />
<br />
A new format is being developed for the DragonBox Pyra handheld. Here are the requirements/notes:<br />
<br />
The compatibility layer should have its own table?<br />
<br />
{| class="wikitable"<br />
|-<br />
! Requirement !! Solutions !! Importance !! Status (decided, undecided, post-proned, won't fix, ...) !! Notes<br />
|-<br />
| Should the new format be supported by Pandora AND Pyra? ||<br />
* Yes<br />
* no<br />
|| Very high || ||<br />
* http://boards.openpandora.org/topic/15530-the-software-side/page-7#entry310691<br />
* This decision affects several of the requirements and possible solutions on this table and should be decided sooner<br />
|-<br />
| API/Daemon to switch between joystick/keyboard mode || || || || Not exactly a PND-error<br />
|-<br />
| Easier to parse than current PNDs and less fragile ||<br />
* UBSON header and metadata + icon + screenshots + cramfs<br />
* tar filesystem<br />
** more future proof<br />
** easier for shell scripts<br />
*** with dd you can extract the tar part and so<br />
** filenames and file permissions are overhead<br />
*** But the overhead is too small to be significant?<br />
** easier to modify after-compiled<br />
*** not really a problem, because the user should be contributing to the recipe anyway<br />
*** a specialized to can be create if another solution is chosen<br />
* zipfs uncompressed filesystem<br />
** See above sections<br />
|| || ||<br />
|-<br />
| Should the metadata include packages dependencies? ||<br />
* Yes<br />
** apt-get install package --print-uris will print download uris for dep packages. Using this uris download packages and extract all (dpkg -x 1.deb ./extracted)<br />
** EvilDragon voted for yes-and-no? http://boards.openpandora.org/topic/15530-the-software-side/page-5#entry310590<br />
*** Or does it mean a minimum "os-version required" metadata on the package and let the user update the system to have the latest libraries available?<br />
** http://boards.openpandora.org/topic/15530-the-software-side/page-5#entry310627<br />
* No.<br />
** EvilDragon commented that he wanna a plug-and-play-and-works system ( http://boards.openpandora.org/topic/15530-the-software-side/page-3#entry310412 )<br />
** If the user is lacking of internet access, "plug-and-play" package won't work<br />
** .deb for general-purpose software<br />
* set of dependencies that system is required to have (SDL, ...)<br />
* Complete user-space like containment features (see http://www.docker.io/ for an example)<br />
|| || Undecided ||<br />
|-<br />
| It should be click-and-run ||<br />
* binfmt<br />
* Sheebang<br />
* File (extension) association<br />
|| || ||<br />
|-<br />
| Less duplication of .desktop and PXML files || || || ||<br />
|-<br />
| AUR-like system:<br />
* Building recipes is easier to replicate and contribute<br />
* If maintainer is jerk, moderator can move his maintanership<br />
|| Fork makepkg || || [http://boards.openpandora.org/topic/15530-the-software-side/page-7#entry310691 Cloudef is working on this task] ||<br />
|-<br />
| Guideline for consistent use of gaming buttons:<br />
* It should reduce the need of patches on top of the ported software<br />
||<br />
* A keyboard/joystick remapping function<br />
* A guideline for software developers<br />
|| || || No single-solution, but a set of them<br />
|-<br />
| A compatibility layer for the current *.PND system:<br />
* The more transparent, the better<br />
||<br />
* Dual-boot<br />
* Virtual Machine<br />
* Tune Debian system to include all libraries used in current SuperZaxxon<br />
** OpenEmbedded it's based on the Debian, afterall<br />
** Probably won't happen because of hard/soft float point library issues<br />
* chroot<br />
** chroot doesn't start services like the real system, then the services of the chroot must be replaced/configured<br />
|| High || ||<br />
* Pyra will have larger internal space, then duplicate system isn't a large problem<br />
* For current OpenPandora, we can already run Debian on SD-card.<br />
** Maybe we can provide Pyra apps for OpenPandora using the Debian-on-sd-card opposed to Debian-on-internal-space<br />
|-<br />
| Cross-building? || || || Undecided ||<br />
|-<br />
| Filesystem type ||<br />
* Cramfs<br />
* Squashfilesystem<br />
|| || ||<br />
|-<br />
| Home-dir of applications ||<br />
* Maybe on per SD-card<br />
* Union-fs?<br />
* overridable by env var?<br />
|| || ||<br />
|-<br />
| Overridable scripts ||<br />
* By creating files with the right name, you can override files in the package (e.g. scripts)<br />
* Yes:<br />
** Cool for hackers<br />
* No:<br />
** Necessitates unionFS.<br />
** Not absolutely necessary - hackers can just rebuild the package.<br />
|| || ||<br />
|-<br />
| File-extension? ||<br />
* .pyra<br />
** We're not using fucking DOS for lord-sake, what's the point of use 3-letters extension?<br />
* .pyd for pyra-data<br />
* .pyr<br />
* .pp for pyra program<br />
* .pa for pyra app<br />
* .drp for dragonbox pyra<br />
** Why limit this to just the Pyra? Could be useful in future systems/elsewhere too.<br />
* ...<br />
|| || undecided. || Maybe a series of polls on the boards?<br />
|-<br />
| Keep what is good || Document all of the good features and port the content to this table || High || Not started ||<br />
|-<br />
| Extra bonus points for a design that can easily be used on other distros and devices as well || || Optional || Optional || Optional<br />
|-<br />
| Multiple binaries/icons per app || || || || Somebody (lost on the sea of comments) didn't like this design, but it should be possible for special software<br />
|-<br />
| Delta packages || || || || http://boards.openpandora.org/topic/15530-the-software-side/page-9#entry311600<br />
|}<br />
<br />
[[Category:PND]]<br />
[[Category:Ideas and proposals]]</div>Vinipsmakerhttps://pandorawiki.org/index.php?title=New_PND_format&diff=29468New PND format2014-02-04T21:42:09Z<p>Vinipsmaker: Clean up the table</p>
<hr />
<div>The current PND format has some shortcomings as listed below. This page should serve as a discussion page/white board for how the format could be improved.<br />
<br />
''' Proposal for discussion, this is just some opening shots '''<br />
<br />
== Current situation ==<br />
The current (ISO-based) PND format has the following shortcomings:<br />
<br />
* It currently uses ISO, SquashFS, and other filesystems... This means that there's no standard to follow, and libpnd needs to carry around support for a multitude of file systems.<br />
* It (often, but not always) uses the [http://en.wikipedia.org/wiki/ISO_9660 ISO file system] which is inefficient at storing data, because:<br />
** It contains duplicate headers for each file entry[http://users.telenet.be/it3.consultants.bvba/handouts/ISO9960.html] (one version with big-endian integers, one version with little-endian integers[http://en.wikipedia.org/wiki/Endianness]) which leads to many kilobytes of wasted space.<br />
** Its file tables are fixed-size, and there are therefore limitations on how many folders you can have, what names you can give your files, how big your files can be etc. For instance, files can only have names that are max 31 characters long, all upper-case, and limited to the ASCII character encoding, and an ISO does only support a folder depth of 8[http://www.on-time.com/rtos-32-docs/rtfiles-32/programming-manual/iso-9660/]. To support more, the Joliet file system extension is needed (or isofs won't recognize all file paths), and the Joliet header can be placed practically anywhere in the file, which means additional seek times.<br />
** It needs various file system extensions to behave correctly, like for example [http://en.wikipedia.org/wiki/Rock_Ridge Rock Ridge Interchange Protocol] and [http://en.wikipedia.org/wiki/Joliet_(file_system) Joliet]. Without them, it becomes pretty useless as demonstrated above, and with them, it becomes difficult to read by a tool (You can only use isofs! There are no other tools out there for programmers to use!).<br />
** It's very difficult to use since you need special ISO making tools to create the image. ISO is a relatively rare format if you're not technically inclined.<br />
** If you want to add or remove files from it, it's near impossible if you use a compact ISO file, since there's no way that you can "expand" an existing ISO file.<br />
* The PND header data is at the end (or in the middle of the file if a screenshot is included) which makes it impossible for e.g. libmagic to recognize the PND file. It will instead recognize it as an ISO file. Having the header data at the end also makes it take a very long time to find the data, making tools and the libpnd library very inefficient.<br />
* The PND file uses a custom XML format for its metadata. There's no reason to do this, especially since the established ".desktop" file format fills exactly the same function.<br />
(note: ".desktop" does not contain all metadata we have wanted over time, but it's of course possible to add extensions à la "X-Pandora-Whatever=xyz")<br />
* There's no "index" for the PND file. The whole file has to be scanned (albeit backwards) to find a PXML file, and there's a big chance for false positives etc.<br />
* Data is just appended linearly to the file so there's no order. If the format is to be extended (to e.g. include an icon file after the screenshot file), should the data just be appended as well?<br />
* UTF-8 is strictly the only encoding that is supported. If you make your PXML on a Windows machine, it won't work.<br />
<br />
== Proposed revisions ==<br />
=== Step 1: File system ===<br />
The file system for PNDs should be replaced by the ''uncompressed'' ZIP archive format. ZIP has the advantage that it's incredibly compact, and uncompressed ZIP makes it possible to read data from the file without having to do any decompression.<br />
<br />
ZIP files are mountable using various implementations of zipfs, and most of these implementations won't store files in memory when they are being read if the ZIP file is uncompressed.<br />
<br />
=== Step 2: Metadata ===<br />
PND files should no longer require special tools for them to be created. Therefore, since ZIP files support random access on files, it shouldn't be necessary to append/prepend metadata to the file. The user simply includes the "PXML.xml" and "preview.png" files inside of the ZIP, and any tools that need information about the PND can simply go to the central directory of the ZIP file (or use a simple ZIP library to do it for them) and get the location of the file inside of the ZIP. This should also dramatically decrease loading times for PNDs.<br />
<br />
Note; performance testing is needed; at the time we started, zip was enormously slower than plain ISO; with driver changes, it may or may not be so, hence our adopted multiple-filesystem-type system. zipfs is one possible option.<br />
<br />
=== Step 3: Metadata format ===<br />
Current PND files include so-called "PXML.xml" files. These files have a custom XML format that has a strange structure.<br />
<br />
These files should be replaced by ".desktop" files. A PND can contain one or more ".desktop" files in its root directory that specify how an application should be launched. PND tools simply use all ".desktop" files they can find in the PND when creating launchers for the contents of the PND.<br />
<br />
== Advantages ==<br />
* There don't have to be ''any'' special tools for reading PND files. The package can be run on any platform using any programming language that can read ZIP files.<br />
* We can use existing facilities to manage the launching of applications. The ".desktop" files can basically be copied without modification into standard locations of the system, and all launchers will become aware of them.<br />
* Reading PND files becomes easier (because of better tool support... sorry, but you can't link to libpnd in all programming languages) and quicker (becuase of the central directory for random file access).<br />
<br />
=== Benchmarks ===<br />
A benchmark was carried through to measure the performance of various package implementations.<br />
The compared systems were:<br />
* ZIP-based uncompressed packages using fuse-zip (a zipfs implementation) to read the files<br />
* ZIP-based compressed packages also using fuse-zip.<br />
* ISO8859-based packages using isofs<br />
* CramFS-based packages.<br />
<br />
A total of 9 files were generated for the test. They have sizes 1, 2, 3, 4, 16, 32, 64, 128 and 256 MB, and were generated via "dd if=/dev/urandom of=$file bs=1M count=$size".<br />
<br />
Because the files contain random data, they did not compress very well. The resulting files were:<br />
* "zipcompressed.zip" 511.1 MB<br />
* "zipuncompressed.zip" 511.0 MB<br />
* "iso.iso" 511.3 MB<br />
* "cramfs.image" 95.4 MB (WOW!)<br />
<br />
Some notes:<br />
* No write tests were done for obvious reasons<br />
* No random access tests were done since the Pandora will use SD cards and thus not get penalized from random access, and that's not what we're interested in anyways.<br />
* Before each test, the commands "sync; echo 3 > /proc/sys/vm/drop_caches" were run.<br />
<br />
==== First test: linear read time ====<br />
Command: "time cat mountpoint/* > /dev/null"<br />
<br />
* ISO: 8.119 seconds<br />
* CramFS: 1.488 seconds (WOW!)<br />
* Zip (compressed): 8.535 seconds<br />
* Zip (uncompressed): 8.290 seconds<br />
<br />
==== Second test: RAM usage ====<br />
-- TODO, have to find a good method of measuring this --<br />
<br />
== Remaining issues ==<br />
* How should libpnd tell the difference between old and new PND files? Should it depend on the "file" tool, should it run its own recognition algorithm, or what? Should the extension be changed to e.g. ".box" (Which was proposed in a thread)?<br />
* How to make sure that the ZIP files are uncompressed? Should we provide a script that "uncompresses" an ordinary ZIP file?<br />
* Is cramfs better than ZIP?<br />
** cramfs cannot be uncompressed like ZIP can<br />
** If you compare compressed ZIP and cramfs, cramfs is more efficient.<br />
** cramfs is harder to use than ZIP (for the developer).<br />
** cramfs ''needs'' to have all of its opened files in memory.<br />
<br />
== Upgrade path from the old PND format ==<br />
A simple script can be written that extracts the old PND file, including the screenshot and the PXML.xml file. The PXML.xml file is then converted to one or many .desktop files, and the desktop files, the preview.png file, and the package contents are compacted into a ZIP that then is renamed to "*.pnd"<br />
<br />
== Usage scenario ==<br />
=== Repackaging of an application from another package format ===<br />
* The user grabs the package for the application<br />
* He dumps the executable and all required libraries into a folder<br />
* He dumps a "screenshot.png" file into the folder that he's made<br />
* He copies the ".desktop" file(s) for the application from the old package, opens them, replaces "Exec=/usr/bin/bla" with "Exec=./bla" and saves them in the directory<br />
* He uses a ZIP archiver to make a ZIP out of the folder, making sure that he sets the "uncompressed" option.<br />
<br />
=== Creating PNDs as part of a build process ===<br />
* The build tool creates a directory with all of the necessary information like above and invokes the "zip" utility to compress the folder.<br />
<br />
=== Accessing a PND's contents for the... ===<br />
<br />
==== ...lazy programmer ====<br />
* The programmer extracts the PND into a folder via libzip and accesses its contents.<br />
<br />
==== ...pragmatic programmer ====<br />
* The programmer uses zipfs to mount the ZIP and accesses the file's contents.<br />
<br />
==== ...smart performance-aware programmer ====<br />
* The programmer fseek()'s the ZIP file until he finds 0x04034b50.<br />
* He jumps 18 bytes forward and reads the int at that location, storing it in "length".<br />
* He jumps 4 bytes and checks if this int matches "length". If it doesn't, it means that the file is compressed, and an error is reported.<br />
* He then jumps forward 4 bytes and stores the short at that location in a variable "nameLength".<br />
* Another 2-byte jump gets the short "extensionsLength".<br />
* He then jumps 2 bytes and reads "nameLength" amount of bytes from the file.<br />
* He then uses strcmp() to see if this string matches the sought-after file. If not, he continues the fseek().<br />
* The programmer now skips "entensionsLength" amount of bytes.<br />
* The programmer reads "length" amount of bytes from the file, and uses this as the sought-after file data.<br />
<br />
==== ...performance fascist/programmer who wants low seek times ====<br />
* The programmer fseek()'s the file from the end until he finds 0x02014b50.<br />
* He then uses the following table to get the information he needs:<br />
{|class="wikitable"<br />
|+ ZIP central directory file header<br />
|-<br />
! Offset !! Bytes !! Description<br />
|-<br />
| 0 || 4 || Central directory file header signature = 0x02014b50<br />
|-<br />
| 4 || 2 || Version made by<br />
|-<br />
| 6 || 2 || Version needed to extract (minimum)<br />
|-<br />
| 8 || 2 || General purpose bit flag<br />
|-<br />
| 10 || 2 || Compression method<br />
|-<br />
| 12 || 2 || File last modification time<br />
|-<br />
| 14 || 2 || File last modification date<br />
|-<br />
| 16 || 4 || CRC-32<br />
|-<br />
| 20 || 4 || Compressed size<br />
|-<br />
| 24 || 4 || Uncompressed size<br />
|-<br />
| 28 || 2 || File name length (''n'')<br />
|-<br />
| 30 || 2 || Extra field length (''m'')<br />
|-<br />
| 32 || 2 || File comment length (''k'')<br />
|-<br />
| 34 || 2 || Disk number where file starts<br />
|-<br />
| 36 || 2 || Internal file attributes<br />
|-<br />
| 38 || 4 || External file attributes<br />
|-<br />
| 42 || 4 || Relative offset of local file header<br />
|-<br />
| 46 || ''n'' || File name<br />
|-<br />
| 46+''n'' || ''m'' || Extra field<br />
|-<br />
| 46+''n''+''m'' || ''k'' || File comment<br />
|}<br />
<br />
* The relative file offset can then be used to jump to the file in question. The file is then scanned as described in the previous section.<br />
<br />
== The 2014/Pyra effort ==<br />
<br />
A new format is being developed for the DragonBox Pyra handheld. Here are the requirements/notes:<br />
<br />
The compatibility layer should have its own table?<br />
<br />
{| class="wikitable"<br />
|-<br />
! Requirement !! Solutions !! Importance !! Status (decided, undecided, post-proned, won't fix, ...) !! Notes<br />
|-<br />
| Should the new format be supported by Pandora AND Pyra? ||<br />
* Yes<br />
* no<br />
|| Very high || ||<br />
* http://boards.openpandora.org/topic/15530-the-software-side/page-7#entry310691<br />
* This decision affects several of the requirements and possible solutions on this table and should be decided sooner<br />
|-<br />
| API/Daemon to switch between joystick/keyboard mode || || || || Not exactly a PND-error<br />
|-<br />
| Easier to parse than current PNDs and less fragile ||<br />
* UBSON header and metadata + icon + screenshots + cramfs<br />
* tar filesystem<br />
** more future proof<br />
** easier for shell scripts<br />
*** with dd you can extract the tar part and so<br />
** filenames and file permissions are overhead<br />
*** But the overhead is too small to be significant?<br />
** easier to modify after-compiled<br />
*** not really a problem, because the user should be contributing to the recipe anyway<br />
*** a specialized to can be create if another solution is chosen<br />
* zipfs uncompressed filesystem<br />
** See above sections<br />
|| || ||<br />
|-<br />
| Should the metadata include packages dependencies? ||<br />
* Yes<br />
** apt-get install package --print-uris will print download uris for dep packages. Using this uris download packages and extract all (dpkg -x 1.deb ./extracted)<br />
** EvilDragon voted for yes-and-no? http://boards.openpandora.org/topic/15530-the-software-side/page-5#entry310590<br />
*** Or does it mean a minimum "os-version required" metadata on the package and let the user update the system to have the latest libraries available?<br />
** http://boards.openpandora.org/topic/15530-the-software-side/page-5#entry310627<br />
* No.<br />
** EvilDragon commented that he wanna a plug-and-play-and-works system ( http://boards.openpandora.org/topic/15530-the-software-side/page-3#entry310412 )<br />
** If the user is lacking of internet access, "plug-and-play" package won't work<br />
** .deb for general-purpose software<br />
* set of dependencies that system is required to have (SDL, ...)<br />
* Complete user-space like containment features (see http://www.docker.io/ for an example)<br />
|| || Undecided ||<br />
|-<br />
| It should be click-and-run ||<br />
* binfmt<br />
* Sheebang<br />
* File (extension) association<br />
|| || ||<br />
|-<br />
| Less duplication of .desktop and PXML files || || || ||<br />
|-<br />
| AUR-like system:<br />
* Building recipes is easier to replicate and contribute<br />
* If maintainer is jerk, moderator can move his maintanership<br />
|| Fork makepkg || || [http://boards.openpandora.org/topic/15530-the-software-side/page-7#entry310691 Cloudef is working on this task] ||<br />
|-<br />
| Guideline for consistent use of gaming buttons:<br />
* It should reduce the need of patches on top of the ported software<br />
||<br />
* A keyboard/joystick remapping function<br />
* A guideline for software developers<br />
|| || || No single-solution, but a set of them<br />
|-<br />
| A compatibility layer for the current *.PND system:<br />
* The more transparent, the better<br />
||<br />
* Dual-boot<br />
* Virtual Machine<br />
* Tune Debian system to include all libraries used in current SuperZaxxon<br />
** OpenEmbedded it's based on the Debian, afterall<br />
** Probably won't happen because of hard/soft float point library issues<br />
* chroot<br />
** chroot doesn't start services like the real system, then the services of the chroot must be replaced/configured<br />
|| High || ||<br />
* Pyra will have larger internal space, then duplicate system isn't a large problem<br />
* For current OpenPandora, we can already run Debian on SD-card.<br />
** Maybe we can provide Pyra apps for OpenPandora using the Debian-on-sd-card opposed to Debian-on-internal-space<br />
|-<br />
| Cross-building? || || || Undecided ||<br />
|-<br />
| Filesystem type ||<br />
* Cramfs<br />
* Squashfilesystem<br />
|| || ||<br />
|-<br />
| Home-dir of applications ||<br />
* Maybe on per SD-card<br />
* Union-fs?<br />
* overridable by env var?<br />
|| || ||<br />
|-<br />
| Overridable scripts ||<br />
* By creating files with the right name, you can override files in the package (e.g. scripts)<br />
* Yes:<br />
** Cool for hackers<br />
* No:<br />
** Necessitates unionFS.<br />
** Not absolutely necessary - hackers can just rebuild the package.<br />
|| || ||<br />
|-<br />
| File-extension? ||<br />
* .pyra<br />
** We're not using fucking DOS for lord-sake, what's the point of use 3-letters extension?<br />
* .pyd for pyra-data<br />
* .pyr<br />
* .pp for pyra program<br />
* .pa for pyra app<br />
* .drp for dragonbox pyra<br />
** Why limit this to just the Pyra? Could be useful in future systems/elsewhere too.<br />
* ...<br />
|| || undecided. || Maybe a series of polls on the boards?<br />
|-<br />
| Keep what is good || Document all of the good features and port the content to this table || High || Not started ||<br />
|-<br />
| Extra bonus points for a design that can easily be used on other distros and devices as well || || Optional || Optional || Optional<br />
|-<br />
| Multiple binaries/icons per app || || || || Somebody (lost on the sea of comments) didn't like this design, but it should be possible for special software<br />
|}<br />
<br />
[[Category:PND]]<br />
[[Category:Ideas and proposals]]</div>Vinipsmakerhttps://pandorawiki.org/index.php?title=New_PND_format&diff=29465New PND format2014-02-03T21:16:47Z<p>Vinipsmaker: adding pyra thoughts/concerns</p>
<hr />
<div>The current PND format has some shortcomings as listed below. This page should serve as a discussion page/white board for how the format could be improved.<br />
<br />
''' Proposal for discussion, this is just some opening shots '''<br />
<br />
== Current situation ==<br />
The current (ISO-based) PND format has the following shortcomings:<br />
<br />
* It currently uses ISO, SquashFS, and other filesystems... This means that there's no standard to follow, and libpnd needs to carry around support for a multitude of file systems.<br />
* It (often, but not always) uses the [http://en.wikipedia.org/wiki/ISO_9660 ISO file system] which is inefficient at storing data, because:<br />
** It contains duplicate headers for each file entry[http://users.telenet.be/it3.consultants.bvba/handouts/ISO9960.html] (one version with big-endian integers, one version with little-endian integers[http://en.wikipedia.org/wiki/Endianness]) which leads to many kilobytes of wasted space.<br />
** Its file tables are fixed-size, and there are therefore limitations on how many folders you can have, what names you can give your files, how big your files can be etc. For instance, files can only have names that are max 31 characters long, all upper-case, and limited to the ASCII character encoding, and an ISO does only support a folder depth of 8[http://www.on-time.com/rtos-32-docs/rtfiles-32/programming-manual/iso-9660/]. To support more, the Joliet file system extension is needed (or isofs won't recognize all file paths), and the Joliet header can be placed practically anywhere in the file, which means additional seek times.<br />
** It needs various file system extensions to behave correctly, like for example [http://en.wikipedia.org/wiki/Rock_Ridge Rock Ridge Interchange Protocol] and [http://en.wikipedia.org/wiki/Joliet_(file_system) Joliet]. Without them, it becomes pretty useless as demonstrated above, and with them, it becomes difficult to read by a tool (You can only use isofs! There are no other tools out there for programmers to use!).<br />
** It's very difficult to use since you need special ISO making tools to create the image. ISO is a relatively rare format if you're not technically inclined.<br />
** If you want to add or remove files from it, it's near impossible if you use a compact ISO file, since there's no way that you can "expand" an existing ISO file.<br />
* The PND header data is at the end (or in the middle of the file if a screenshot is included) which makes it impossible for e.g. libmagic to recognize the PND file. It will instead recognize it as an ISO file. Having the header data at the end also makes it take a very long time to find the data, making tools and the libpnd library very inefficient.<br />
* The PND file uses a custom XML format for its metadata. There's no reason to do this, especially since the established ".desktop" file format fills exactly the same function.<br />
(note: ".desktop" does not contain all metadata we have wanted over time, but it's of course possible to add extensions à la "X-Pandora-Whatever=xyz")<br />
* There's no "index" for the PND file. The whole file has to be scanned (albeit backwards) to find a PXML file, and there's a big chance for false positives etc.<br />
* Data is just appended linearly to the file so there's no order. If the format is to be extended (to e.g. include an icon file after the screenshot file), should the data just be appended as well?<br />
* UTF-8 is strictly the only encoding that is supported. If you make your PXML on a Windows machine, it won't work.<br />
<br />
== Proposed revisions ==<br />
=== Step 1: File system ===<br />
The file system for PNDs should be replaced by the ''uncompressed'' ZIP archive format. ZIP has the advantage that it's incredibly compact, and uncompressed ZIP makes it possible to read data from the file without having to do any decompression.<br />
<br />
ZIP files are mountable using various implementations of zipfs, and most of these implementations won't store files in memory when they are being read if the ZIP file is uncompressed.<br />
<br />
=== Step 2: Metadata ===<br />
PND files should no longer require special tools for them to be created. Therefore, since ZIP files support random access on files, it shouldn't be necessary to append/prepend metadata to the file. The user simply includes the "PXML.xml" and "preview.png" files inside of the ZIP, and any tools that need information about the PND can simply go to the central directory of the ZIP file (or use a simple ZIP library to do it for them) and get the location of the file inside of the ZIP. This should also dramatically decrease loading times for PNDs.<br />
<br />
Note; performance testing is needed; at the time we started, zip was enormously slower than plain ISO; with driver changes, it may or may not be so, hence our adopted multiple-filesystem-type system. zipfs is one possible option.<br />
<br />
=== Step 3: Metadata format ===<br />
Current PND files include so-called "PXML.xml" files. These files have a custom XML format that has a strange structure.<br />
<br />
These files should be replaced by ".desktop" files. A PND can contain one or more ".desktop" files in its root directory that specify how an application should be launched. PND tools simply use all ".desktop" files they can find in the PND when creating launchers for the contents of the PND.<br />
<br />
== Advantages ==<br />
* There don't have to be ''any'' special tools for reading PND files. The package can be run on any platform using any programming language that can read ZIP files.<br />
* We can use existing facilities to manage the launching of applications. The ".desktop" files can basically be copied without modification into standard locations of the system, and all launchers will become aware of them.<br />
* Reading PND files becomes easier (because of better tool support... sorry, but you can't link to libpnd in all programming languages) and quicker (becuase of the central directory for random file access).<br />
<br />
=== Benchmarks ===<br />
A benchmark was carried through to measure the performance of various package implementations.<br />
The compared systems were:<br />
* ZIP-based uncompressed packages using fuse-zip (a zipfs implementation) to read the files<br />
* ZIP-based compressed packages also using fuse-zip.<br />
* ISO8859-based packages using isofs<br />
* CramFS-based packages.<br />
<br />
A total of 9 files were generated for the test. They have sizes 1, 2, 3, 4, 16, 32, 64, 128 and 256 MB, and were generated via "dd if=/dev/urandom of=$file bs=1M count=$size".<br />
<br />
Because the files contain random data, they did not compress very well. The resulting files were:<br />
* "zipcompressed.zip" 511.1 MB<br />
* "zipuncompressed.zip" 511.0 MB<br />
* "iso.iso" 511.3 MB<br />
* "cramfs.image" 95.4 MB (WOW!)<br />
<br />
Some notes:<br />
* No write tests were done for obvious reasons<br />
* No random access tests were done since the Pandora will use SD cards and thus not get penalized from random access, and that's not what we're interested in anyways.<br />
* Before each test, the commands "sync; echo 3 > /proc/sys/vm/drop_caches" were run.<br />
<br />
==== First test: linear read time ====<br />
Command: "time cat mountpoint/* > /dev/null"<br />
<br />
* ISO: 8.119 seconds<br />
* CramFS: 1.488 seconds (WOW!)<br />
* Zip (compressed): 8.535 seconds<br />
* Zip (uncompressed): 8.290 seconds<br />
<br />
==== Second test: RAM usage ====<br />
-- TODO, have to find a good method of measuring this --<br />
<br />
== Remaining issues ==<br />
* How should libpnd tell the difference between old and new PND files? Should it depend on the "file" tool, should it run its own recognition algorithm, or what? Should the extension be changed to e.g. ".box" (Which was proposed in a thread)?<br />
* How to make sure that the ZIP files are uncompressed? Should we provide a script that "uncompresses" an ordinary ZIP file?<br />
* Is cramfs better than ZIP?<br />
** cramfs cannot be uncompressed like ZIP can<br />
** If you compare compressed ZIP and cramfs, cramfs is more efficient.<br />
** cramfs is harder to use than ZIP (for the developer).<br />
** cramfs ''needs'' to have all of its opened files in memory.<br />
<br />
== Upgrade path from the old PND format ==<br />
A simple script can be written that extracts the old PND file, including the screenshot and the PXML.xml file. The PXML.xml file is then converted to one or many .desktop files, and the desktop files, the preview.png file, and the package contents are compacted into a ZIP that then is renamed to "*.pnd"<br />
<br />
== Usage scenario ==<br />
=== Repackaging of an application from another package format ===<br />
* The user grabs the package for the application<br />
* He dumps the executable and all required libraries into a folder<br />
* He dumps a "screenshot.png" file into the folder that he's made<br />
* He copies the ".desktop" file(s) for the application from the old package, opens them, replaces "Exec=/usr/bin/bla" with "Exec=./bla" and saves them in the directory<br />
* He uses a ZIP archiver to make a ZIP out of the folder, making sure that he sets the "uncompressed" option.<br />
<br />
=== Creating PNDs as part of a build process ===<br />
* The build tool creates a directory with all of the necessary information like above and invokes the "zip" utility to compress the folder.<br />
<br />
=== Accessing a PND's contents for the... ===<br />
<br />
==== ...lazy programmer ====<br />
* The programmer extracts the PND into a folder via libzip and accesses its contents.<br />
<br />
==== ...pragmatic programmer ====<br />
* The programmer uses zipfs to mount the ZIP and accesses the file's contents.<br />
<br />
==== ...smart performance-aware programmer ====<br />
* The programmer fseek()'s the ZIP file until he finds 0x04034b50.<br />
* He jumps 18 bytes forward and reads the int at that location, storing it in "length".<br />
* He jumps 4 bytes and checks if this int matches "length". If it doesn't, it means that the file is compressed, and an error is reported.<br />
* He then jumps forward 4 bytes and stores the short at that location in a variable "nameLength".<br />
* Another 2-byte jump gets the short "extensionsLength".<br />
* He then jumps 2 bytes and reads "nameLength" amount of bytes from the file.<br />
* He then uses strcmp() to see if this string matches the sought-after file. If not, he continues the fseek().<br />
* The programmer now skips "entensionsLength" amount of bytes.<br />
* The programmer reads "length" amount of bytes from the file, and uses this as the sought-after file data.<br />
<br />
==== ...performance fascist/programmer who wants low seek times ====<br />
* The programmer fseek()'s the file from the end until he finds 0x02014b50.<br />
* He then uses the following table to get the information he needs:<br />
{|class="wikitable"<br />
|+ ZIP central directory file header<br />
|-<br />
! Offset !! Bytes !! Description<br />
|-<br />
| 0 || 4 || Central directory file header signature = 0x02014b50<br />
|-<br />
| 4 || 2 || Version made by<br />
|-<br />
| 6 || 2 || Version needed to extract (minimum)<br />
|-<br />
| 8 || 2 || General purpose bit flag<br />
|-<br />
| 10 || 2 || Compression method<br />
|-<br />
| 12 || 2 || File last modification time<br />
|-<br />
| 14 || 2 || File last modification date<br />
|-<br />
| 16 || 4 || CRC-32<br />
|-<br />
| 20 || 4 || Compressed size<br />
|-<br />
| 24 || 4 || Uncompressed size<br />
|-<br />
| 28 || 2 || File name length (''n'')<br />
|-<br />
| 30 || 2 || Extra field length (''m'')<br />
|-<br />
| 32 || 2 || File comment length (''k'')<br />
|-<br />
| 34 || 2 || Disk number where file starts<br />
|-<br />
| 36 || 2 || Internal file attributes<br />
|-<br />
| 38 || 4 || External file attributes<br />
|-<br />
| 42 || 4 || Relative offset of local file header<br />
|-<br />
| 46 || ''n'' || File name<br />
|-<br />
| 46+''n'' || ''m'' || Extra field<br />
|-<br />
| 46+''n''+''m'' || ''k'' || File comment<br />
|}<br />
<br />
* The relative file offset can then be used to jump to the file in question. The file is then scanned as described in the previous section.<br />
<br />
== The 2014/Pyra effort ==<br />
<br />
A new format is being developed for the DragonBox Pyra handheld. Here are the requirements/notes (this table should have an importance cell and a status (decided, undecided, post-proned, ...) cell):<br />
<br />
The compatibility layer should have its own table?<br />
<br />
{| class="wikitable"<br />
|-<br />
! Requirement !! Solutions !! Notes<br />
|-<br />
| API/Daemon to switch between joystick/keyboard mode || || Not exactly a PND-error<br />
|-<br />
| Easier to parse than current PNDs and less fragile ||<br />
* UBSON header and metadata + icon + screenshots + cramfs<br />
* tar filesystem<br />
** more future proof<br />
** easier for shell scripts<br />
*** with dd you can extract the tar part and so<br />
** filenames and file permissions are overhead<br />
*** But the overhead is too small to be significant?<br />
** easier to modify after-compiled<br />
*** not really a problem, because the user should be contributing to the recipe anyway<br />
*** a specialized to can be create if another solution is chosen<br />
||<br />
|-<br />
| Should the metadata include packages dependencies? ||<br />
* Yes<br />
** apt-get install package --print-uris will print download uris for dep packages. Using this uris download packages and extract all (dpkg -x 1.deb ./extracted)<br />
** EvilDragon voted for yes-and-no? http://boards.openpandora.org/topic/15530-the-software-side/page-5#entry310590<br />
*** Or does it mean a minimum "os-version required" metadata on the package and let the user update the system to have the latest libraries available?<br />
** http://boards.openpandora.org/topic/15530-the-software-side/page-5#entry310627<br />
* No.<br />
** EvilDragon commented that he wanna a plug-and-play-and-works system ( http://boards.openpandora.org/topic/15530-the-software-side/page-3#entry310412 )<br />
** If the user is lacking of internet access, "plug-and-play" package won't work<br />
** .deb for general-purpose software<br />
* set of dependencies that system is required to have (SDL, ...)<br />
* Complete user-space like containment features (see http://www.docker.io/ for an example)<br />
|| Undecided<br />
|-<br />
| It should be click-and-run ||<br />
* binfmt<br />
* Sheebang<br />
* File (extension) association<br />
||<br />
|-<br />
| Less duplication of .desktop and PXML files || || <br />
|-<br />
| AUR-like system:<br />
* Building recipes is easier to replicate and contribute<br />
* If maintainer is jerk, moderator can move his maintanership<br />
|| Fork makepkg || <br />
|-<br />
| Guideline for consistent use of gaming buttons:<br />
* It should reduce the need of patches on top of the ported software<br />
||<br />
* A keyboard/joystick remapping function<br />
* A guideline for software developers<br />
|| No single-solution, but a set of them<br />
|-<br />
| A compatibility layer for the current *.PND system:<br />
* The more transparent, the better<br />
||<br />
* Dual-boot<br />
* Virtual Machine<br />
* Tune Debian system to include all libraries used in current SuperZaxxon<br />
** OpenEmbedded it's based on the Debian, afterall<br />
** Probably won't happen because of hard/soft float point library issues<br />
* chroot<br />
** chroot doesn't start services like the real system, then the services of the chroot must be replaced/configured<br />
||<br />
* Pyra will have larger internal space, then duplicate system isn't a large problem<br />
* For current OpenPandora, we can already run Debian on SD-card.<br />
** Maybe we can provide Pyra apps for OpenPandora using the Debian-on-sd-card opposed to Debian-on-internal-space<br />
|-<br />
| Cross-building? || || Undecided<br />
|-<br />
| Filesystem type ||<br />
* Cramfs<br />
* Squashfilesystem<br />
||<br />
|-<br />
| Home-dir of applications ||<br />
* Maybe on per SD-card<br />
* Union-fs<br />
* overridable by env var?<br />
||<br />
|-<br />
| File-extension? ||<br />
* .pyra<br />
** We're not using fucking DOS for lord-sake, what's the point of use 3-letters extension?<br />
* .pyd for pyra-data<br />
* .pyr<br />
* .pp for pyra program<br />
* .pa for pyra app<br />
* .drp for dragonbox pyra<br />
* ...<br />
|| undecided. Maybe a series of pools on the boards?<br />
|-<br />
| Keep what is good || Document all of the good features and port the content to this table || Not started<br />
|-<br />
| Extra bonus points for a design that can easily be used on other distros and devices as well || || Optional<br />
|-<br />
| Multiple binaries/icons per app || || Somebody (lost on the sea of comments) didn't like this design, but it should be possible for special software<br />
|}<br />
<br />
[[Category:PND]]<br />
[[Category:Ideas and proposals]]</div>Vinipsmaker