Extend Utils

From Pandora Wiki
Revision as of 21:58, 17 July 2010 by Stuckie (talk | contribs) (Lots of documentation on the new Extend Utils suite.)
Jump to: navigation, search

Overview

Extend Utils provide a variety of tools to manipulate Extend, Overlay and Swap files. These are generally loop-back files stored on SD card for various purposes which include, but is not limited to:

  • OS Extends - running another distribution on top of Angstrom in a chroot environment.
  • Dev Extends - a method of providing dev tool support without installing directly to NAND.
  • File system Overlays - redirect writes from NAND to a file on an SD card.
  • Swap - use a swap file on SD card to provide additional memory for the Pandora when needed.

This page used to only document OS Extends... it will now document all facets of the Extend Utils.

Extend Utils have been written by Stuckie and while very much work-in-progress, the scripts are very useful in day to day tasks.

The Extend Utils suite is updated regularly, and will always be available here, as part of the SimpleDev system: Extend Utils

For support, please direct questions to the relevant threads on the forum - see Links at the bottom of this page.

Extend Utils Help

Extend Utils ( version 1.0a ) now facilitates the creation of "PreMount" scripts. For users of previous Extend Utils script sets, these separate scripts are essentially a PreMount script - the Ubuntu specific variant being a prime example.

This allows us to create customised mounting procedures - such as being able to mount multiple Extends and Overlays at once; for example, a compressed Base OS Extend with perhaps an Overlay on top to "catch" installed programs.

Unfortunately, as is always the case with more customization, the complexity has shot up. This guide will help you to create a PreMount script given four common scenarios: a Swap mount, a Home or Root Overlay, an OS Extend and a Dev Extend.

Creating a Swap PreMount

Perhaps the simplest PreMount script is a Swap Mount. For this, I shall assume you have not created a .swap file.

  • Start up Extend Utils.
  • Select New Extend/Overlay/Swap.
  • Select New Swap File.
  • Select 256MB - this uses one of the pre-created zipped Extend files, whereas Custom will create a new one on the card.
  • Save the file anywhere you like.
  • It will extract the appropriate zip to where you selected.
  • Back at the Main Menu, select New Pre-Mount.
  • Select New Swap Mount Script.
  • Find your Swap File you just created.
  • Select a location for your Pre-Mount script to be saved to.
  • Finally, back at the Main Menu, you can select Mount Existing Pre-Mount.
  • Find where your Pre-Mount script was saved to, and that's it.. you have swap mounted.

Obviously, now that the script is saved, you only need to repeat the last two steps, or just run the script direct from the Terminal.

Creating a Root or Home Overlay PreMount

Again, I shall assume that you do not have any pre-made .overlay files.

  • Start up Extend Utils.
  • Select New Overlay.
  • Select New Overlay File.
  • Select 1GB - or whatever you like.. again be aware that Custom may very well chew through your SD card as it DD's direct to it!
  • Save the file anywhere you like.
  • It will extract the appropriate zip to where you selected.
  • Back at the Main Menu, select New Pre-Mount.
  • Select New Overlay Mount Script.
  • Select either Standard Home Overlay or Standard Root Overlay.
  • Find your Overlay that you created.
  • Select where you want to save the Pre-Mount script.
  • Finally, back at the Main Menu, you can select Mount Existing Pre-Mount.
  • Find where your Pre-Mount script was saved to, and that's it.. either the majority of / or your /home is bound to the overlay.

Creating an OS Extend PreMount

Generally, these will have been pre-made downloads, or you'll have created a chroot environment and wrapped it in an extend already. It's also important to note that there are essentially TWO types of OS Extends - compressed and uncompressed. Compressed OS Extends should be affixed with compressed; base-debian-sid-compressed.extend for example, which donates it's a base install of debian sid, in a compressed extend. Compressed extends can only be mounted as read-only, and will likely fail if you try to mount them read-write. This means they require an Overlay in order to be much use. As such, I will describe a "classic" uncompressed OS Extend procedure, and a "modern" compressed OS Extend procedure.

Compressed OS Extends

  • Start Extend Utils.
  • Select New Pre-Mount.
  • Select New Extend Mount.
  • Select OS Extend.

This is where you need to start paying attention as to what you're doing. For our purposes, we want a basic configuration for say a Base Debian Sid Extend, and one Overlay. The Base Debian Sid Extend we would have downloaded already, and the Overlay we should have created as well.

  • Move the slider to select 1 Read-Only Extend.
  • Select our base-debian-sid-compressed.extend ( assuming you've downloaded it, and it's been released! )
  • We don't want any Read-Write Extends, so Cancel or select 0.
  • We do want an Overlay, so select 1 Read-Write Overlay.
  • Choose the Overlay you want to use.
  • Now you must name the mount point. This can be anything you like, but it must be unique and not the name of the Extend or Overlay you've chosen.
  • You'll then be asked where you want to save the PreMount script.

Again, we need to pay attention as to what we want here. For our purposes, we just want a Terminal, so that we can call apt-get and install whatever we feel like, and run it from the shell to interact with the program in Angstrom. You can, of course, re-do this procedure later after having installed a window manager and desktop manager ( IE: something like Fluxbox and GDM ) so that you can jump fully into Debian if you so wish.

  • Select the Terminal option.
  • You'll now be sent back to the Main Menu, where you can now select Mount Existing Pre-Mount.
  • Select your Pre-Mount Script you've just created.
  • After the gksudo checks, you should have a Terminal open inside your OS Extend, where all writes go to your Overlay.

Uncompressed/Classic OS Extends

These ones are much easier to deal with, and are generally what have been released up till now. These don't require an Overlay ( though you can still use one if you like. ) The procedure for setting up a mount script for one of these runs as follows:

  • Start Extend Utils.
  • Select New Pre-Mount.
  • Select New Extend Mount.
  • Select OS Extend.

As said, we don't need an Overlay, and Uncompressed/Classic Extends can be mounted Read-Write without any issues. Therefore, we only need to select 1 Read-Write Extend, and 0 everything else.

  • We don't want any Read-Only Extends, so Cancel or select 0.
  • We don want a Read-Write Extend...so move the slider to 1 and find your Extend file.
  • Again, we don't want an Overlay, so select 0 or Cancel it.
  • Now you must name the mount point. This can be anything you like, but it must be unique and not the name of the Extend.
  • You'll then be asked where you want to save the PreMount script.

Now we need to know roughly what's in our Extend.. to be safe, we can just choose a Terminal as that's guaranteed to work. If, however, you know that GDM is installed and setup ( I'll ensure that any of my Extends that are, are documented in saying so ) then you can choose the GDM option.

  • We'll assume we've got GDM present, so select GDM.
  • You'll now be sent back to the Main Menu, where you can now select Mount Existing Pre-Mount.
  • Select your Pre-Mount Script you've just created.
  • After the gksudo checks, GDM will start up alongside Angstrom and ask you to login. You should have a login/password already - either from you manually setting it up, or the default that whoever set the .Extend up performed.

Creating a Dev Extend Pre-Mount

Dev Extends are essentially Root Extends these days.. however that's not to say there won't eventually be packs so that there's a base Dev Extend with just the bare essentials, then addons for GTK, QT, wxWidgets, etc.. so having them classed as an Extend seems like the best option.

The current Dev Extends are created exactly like the Uncompressed/Classic OS Extends, but for completeness sakes:

  • Start Extend Utils.
  • Select New Pre-Mount.
  • Select New Extend Mount.
  • Select Dev Extend.
  • Select 0 Read-Only Extends.
  • Select 1 Read-Write Extend.
  • Find your Dev Extend.
  • Select 0 Read-Write Overlays.
  • Come up with a mount point.
  • Save your Pre-Mount somewhere.
  • You may now load it up through the Mount Existing Pre-Mount script if you so wish.

OS Extends

OS Extend is a method of running another distribution on top of Angstrom. This is currently very much a hacky WORK IN PROGRESS method of doing so, and this page describes what it does, how to get started, and any major pitfalls in using it.

Currently, I've been successful in getting Debian, Ubuntu and Arch running to varying degrees. This page will be split up into these distributions so you should be able to jump to the correct section for any oddities that I've uncovered, and any workarounds presented.

Essentially, what I'm doing is using a loop back file, formatted as Ext2, and performing a first stage install within it for Arm. This is called by either debootstrap, rootstock, or whichever system the target distribution uses. These "Extend" files are then copied over to SD card/Memory Stick/etc... and chrooted into.

However, this brings some very important caveats:

  • System Services don't always tend to work, and starting them can clobber Angstrom's services ( see the HAL/DBus issues in Ubuntu below. )
  • Limited resources when running TWO XServers and TWO Desktop Managers does start to show, but the fact that it's still running fairly well is a testament to how powerful the Pandora actually is.
  • Logging out of XFCE will speed up the chrooted environment, but tends to shut down NetworkManager with it, disabling WiFi for the environment unless it knows how to handle it.

Hopefully, these can be dealt with in due time.

That said, there are benefits of using a system such as this:

  • It's (almost) the exact same distribution as you'd be installing on SD Card, so why not test it out first before sacrificing an SD card fully?
  • Perhaps you only have the one SD card? It may be a gigantic SD card but partitioning scares you, so this allows you to try a chosen distribution first without messing about!
  • Some distributions have a HUGE library of packages ready and waiting to be played with.. why not see which ones you'd like ported over to Angstrom natively?
  • We can be sneaky and take advantage of all Pandora specific updates to Angstrom, and have our Extended OS install reap the benefits rather than having to wait for someone to port it over ( kernel drivers, for example. )
  • Why not show off just how powerful the Pandora actually is - you are running TWO X Servers and TWO Desktop Managers at once!
  • Development between multiple distributions can be done much more easily, as you could bind your development distribution, and check driver support in Angstrom.
  • And my absolute favourite - it's a fun hack!

So, there is good reason to try and pursue a system such as this, if perhaps not generally geared to "End User Consumption."

Arch OS Extend

A proof of concept is available here: 1Gb Arch Extend

Not much work has been taking in furthering the install as of this time.

Android OS Extend

This might be tricky due to dalvik.. doesn't seem to be any fast way of "borrowing" an existing rootfs, will have to just try compiling manually for the Pandora.

Links

Debian OS Extend

Debian was the first distribution to work in this environment.

Various proof of concepts currently exist:

Work on Debian has currently active...ish.

If you get "cannot open display: :0.0" this is a regression in the script for xhost support, please run the following in an Angstrom terminal: xhost local:localhost

Sorry about that!

Ubuntu OS Extend

Ubuntu development is on hold.

A first release is available here:

Ubuntu OS Extend currently uses a slightly tailored version of Extend Utils, available here: Ubuntu Extend

Ubuntu can be started via the Start Ubuntu option in the System Menu. Then just follow the on-screen prompts.

F.A.Q

Q: How do I login?
A: The default user is "ubuntu" with password "pandora"

Q: Can I create a new user?
A: Yes, but you'll have to do it manually from the Terminal, or start the DBus service ( see below. )

Q: Can I upgrade to Karmic or Lucid?
A: I'd highly advise against this due to the usual System V init system being replace by Upstart, which isn't too happy about being wrapped in a chroot.

Issues

Ubuntu seems to be very dependent upon it's system services being operational. This unfortunately causes us problems as we're jumping in via a chroot, rather than booting the system normally, so Ubuntu's services haven't started; though Angstrom's have.

It is possible to get Ubuntu's services to start by calling sudo /etc/init.d/dbus start however, this knocks out Angstrom's services, and in particular, HAL - the Hardware Abstraction Layer - which currently detects the presence of a chroot and refuses to work inside our environment. This is bad as if we log out of Ubuntu, we've lost access to all input devices, unless we have switched back to Angstrom and restarted dbus in it's environment.

Update: Hal can be kept active by following this routine...

  • Start Ubuntu as normal, but don't click the final Ok for GDM to start.
  • Open a Terminal
  • sudo chroot /tmp/osextend
  • Open another Terminal/Tab ... this'll be Angstrom, whereas your original is Ubuntu.
  • sudo /etc/init.d/dbus-1 stop (Angstrom)
  • sudo /etc/init.d/dbus start (Ubuntu)
  • sudo hald --daemon=no --verbose=yes (Angstrom)
  • Wait till HAL has finished, then click OK.

This gets us slightly further, but still seems to stop Ubuntu's admin services somewhere, as you still cannot use the GUI-based Administration tools. You do however, seem to have access to WiFi and Bluetooth as their icons now appear in the system tray. D-Pad control now seems knocked out as well.

Touchscreen support currently does not work. The TSLib has been installed, and the xorg-input-tslib is also available, however it doesn't seem to be picked up.

Due to Ubuntu's services not being active, most GUI-driven Administration utilities do not appear to work. Either perform the manual command line work, or start services as above.

If Angstrom's X is quit ( IE: you log out of XFCE, ) although a speed up is gained, you've lost access to WiFi. I haven't yet checked if starting services inside Ubuntu would allow the setting up of the WiFi from inside the OS Extend.

The CTRL+ALT+F# keys for changing virtual terminals work on a USB keyboard in Ubuntu, but not on the Pandora's keyboard; neither work in Angstrom.. as such, you have to perform sudo chvt # replacing # with 2 for Ubuntu, and 7 for Angstrom.

Ubuntu 9.04 - Jaunty Jackalope, is currently the last version of Ubuntu to work "properly" in a chroot due to later versions changing the init deamon from the standard System V to Upstart.

Further Work

Ubuntu has been put on hold for the time being.

I also need to have a further look into getting Karmic or Lucid running properly.

Links

Dev Extend

A Dev Extend is a special kind of Overlay which only binds enough of the file system to allow development tools to work. As such, most information on Overlays - particularly Root Overlays - match Dev Extends.

Dev Extends are routinely updated and match Stuckie's current development environment. Support for updating between revisions is planned, but not currently implemented.

Overlays

Overlays allow the base system to be kept unmodified, and all changes to be propagated to the Overlay file. This is immensely useful for development work where you can install a development environment "normally" and compile what is needed, then unbind the overlay and run it on a "vanilla" install, to see if any libraries are missing.

Stuckie's general development environment uses a Dev Extend and an Overlay - so that all compiled code sits in the Home Overlay, and the development environment is in the Dev Extend. This allows any programs that have not been configured fully to keep clear of the NAND when running, should they create config files in the user's Home directory, as well as separating the compiled programs from the development environment for easier testing on the "vanilla" state.

Overlays essentially come in two flavours - Home and Root - where the latter covers everything written to / and the former only covering /home, however an Overlay could be made for any combination of directories.

Extend Utils GUI

On the forums, SomeGuy99 has suggested the creation of a GUI-based app to wrap around the Extend Utils suite. This would greatly simplify what is going on by being able to provide on-screen help, as well as tying the many scripts into one app and not cluttering up the System menu any further.

This also needs to be a cross-platform application, if only for the creation of Extend files. As such, it needs a cross-platform widget toolkit; preferably a light-weight one to run on Pandora as well. Currently, this looks to be FLTK

For the time being, the Extend Utils has been rewritten ( version 1.0a ) to further use Zenity and create customised mount scripts, in preparation for the work on the FLTK-based GUI.

Links

Retrieved from "https://pandorawiki.org/index.php?title=Extend_Utils&oldid=3042"