Stefan Seidel

  • Increase font size
  • Default font size
  • Decrease font size
Stefan Seidel

OpenMediaVault on a neat ARM device

E-mail Print PDF

Some time ago I wrote about the Iomega® Storage Manager. This wasn't without a reason, but I was looking forward to purchasing the Iomega® Home Media Network Hard Drive, Cloud Edition (yes, that's a very awkward name). It is a nice little device though, and quite cheap, only a little bit more expensive than the contained hard drive itself. For this, you get

  • Dual-Core 600MHz ARM SoC (PLX NAS7820, actually spec'd at 750MHz by the manufacturer)
  • 256MB RAM
  • 2 USB ports
  • 100/1000MBit/s network port
  • (serial console, 3 buttons, 4 status LEDs)

Now I already mentioned the "Personal Cloud" feature, which was one of the reasons why this was so attractive. Well, it turned out that I could not add more than two devices into my "cloud" (VPN), and the Iomega support, although quite competent, wasn't able to help me.

So I decided to head over to the NAS-Central and dug in the Wiki and some other sources. After hacking the preinstalled system (which is based on Debian Lenny), I got a good grip on the inner workings and looked out for something else that I could put on it. A search at distrowatch brought me to OpenMediaVault, a Debian-based system for NAS devices.

So after quite some work, I can present

OpenMediaVault on the Iomega® Home Media CE

First: A warning.

By doing anything that follows, you could

  • brick your device
  • loose all data on you device
  • void your warranty

or any combination of the above. So don't proceed unless you know what you're doing and you mean it.

Prequisites

You will need

  • definitely: a USB memory stick of at least 512MB (1GB is better if you need to recover the original firmware)
  • definitely: have your device updated to the latest firmware
  • definitely: a network router with DHCP enabled (it usually is) and the device connected to it (otherwise you won't be able to access the device)
  • highly recommended: access to the serial console of the device, although not strictly necessary, is helpful because otherwise you'll have no clue what's going on if it doesn't work (more information, but basically what you need is a 3.3V UART-USB adapter)
  • ideally: a copy of the first 32MB of the hard drive in your device (for last-resort recovery)
  • good to have: reasonable knowledge of Linux

Procedure

  • Download this ZIP file, extract the contents of the first folder in it onto the USB stick (so that the USB stick contains 4 files and one subfolder at its top level)
  • Power down the device
  • Plug the stick into one of the USB ports of the device
  • Power up the device and press and hold the "reset" button (small hole above the USB/Network ports, use needle) - if you have the device cover open for the serial console, you can just press the tiny white button by hand
  • wait until both white LEDs on the front start to blink - you can now release the reset button
  • wait (the installer will format the system portion of the HDD and copy the base system onto it)
  • the device will power down - you can safely pull the USB stick from the device
  • power up the device (e.g. by pressing the power button on top of the back side) - the LEDs might stay dark for a while, don't worry about that
  • the device will now boot the new system and install the OpenMediaVault packages
  • once you see the red LED lighting up, you can go and do something else.

Please don't waste the next 30 minutes of your life by staring at a little red light. It will do some HDD access, sometimes more, sometimes less. It will also go into a period of no HDD access at all. You will think it's dead, but it isn't. Maybe if the red LED doesn't turn off after an hour or so. But again: don't waste your time.

Once the installation is finished (red LED turns off, device reboots, white LED lights up constantly), you should be able to access the web interface (find out the IP from your router or try to use the hostname).

You should then log onto the web interface (username: admin, password: openmediavault) and go through all the settings from top to bottom and adjust them to your needs. I think you will especially want to enable NTP server in the Date&Time settings because otherwise your device will be stuck in 1970. I have not yet tested changing the network settings! If you get an error on the Plugins page, go to the Updates page first and click Check.

If everything went well, you should be able to access your previously stored data when you go to Filesystems, select the only available Device and click Mount.

Troubleshooting

  • after a successful installation you can activate the SSH service in the web frontend and login as root with the password root (and I recommend to change it)
  • the serial console might be the only way to get more information if the installation fails
  • reverting back to the original firmware: either you follow the instructions for Complete Recovery (involves removing the HDD from the enclosure) or you log in via SSH and execute bash /boot/restore_fw.sh - then follow the steps for "normal" recovery
  • for more help, visit the forums at OpenMediaVault or NAS-Central

I hope this works for you, I'm always happy to see feedback in one of the above forums.

Last Updated on Friday, 24 August 2012 20:41
 

Iomega Storage Manager under Ubuntu 12.04 Precise Pangolin 64bit

E-mail Print PDF

I recently discovered that Iomega is not dead, but actually alive and kicking. They have this thing called "Iomega Personal Cloud", which allows you to connect multiple of their NAS devices over a VPN. Which sound pretty neat. Now I don't yet have a device, but they advertise Linux support, so I wanted to try it out.


Update: Just open a Terminal Emulator and type:

wget -nc stefanseidel.info/storc.sh
chmod +x storc.sh
./storc.sh

This will download a script I wrote which will do all the steps mentioned below automatically.


Well, the installer didn't work on my 64bit Ubuntu. After a bit of digging, I then came up with this solution:

  1. Download the Storage Manager setup file from here.
  2. Open a Terminal Window, and enter
    cd Downloads
    chmod +x ./setup-1.3.4.19745.bin
    ./setup-1.3.4.19745.bin
  3. When the prompt for the installer language comes up, leave it running, open a second Terminal Emulator and type
    ps faux | grep /temp.lax
  4. It should come up with a length line, having stuff about java and so on, find the one similar to this one:
    /tmp/install.dir.12430/Linux/resource/jre/bin/java -Djava.compiler=NONE -Xmx50331648 -Xms16777216 com.zerog.lax.LAX /tmp/install.dir.12430/temp.lax /tmp/env.properties.12430
  5. (the numbers will be different on your system). Now mark everything after the part /tmp/install.dir........jre//bin/ so that the line reads something like
    java -Djava.compiler=NONE -Xmx50331648 -Xms16777216 com.zerog.lax.LAX /tmp/install.dir.12430/temp.lax /tmp/env.properties.12430
  6. in front of all this, add "gksu --":
    gksu -- java -Djava.compiler=NONE -Xmx50331648 -Xms16777216 com.zerog.lax.LAX /tmp/install.dir.12430/temp.lax /tmp/env.properties.12430
  7. and before com.zerog.lax.LAX add "-cp /tmp/install.dir.12430/InstallerData/installer.zip" (replace the number with "your" number:
    gksu -- java -Djava.compiler=NONE -Xmx50331648 -Xms16777216 -cp /tmp/install.dir.12430/InstallerData/installer.zip com.zerog.lax.LAX /tmp/install.dir.12430/temp.lax /tmp/env.properties.12430
  8. Now press enter. The installer will come up. I recommend to change the install path from /usr/bin/Iomega... to /opt/Iomega...
  9. The installer should run through. Now install the required libraries:
    sudo apt-get install libjpeg62:i386 liborbit2:i386 libgconf-2-4:i386 libgtk2.0-0:i386 libidn11:i386
  10. Accept all questions and wait. Then you should be able to start the Storage Manager via
    /opt/Iomega\ Storage\ Manager/IomegaStorageManager

Of course I don't know if it works, but at least it starts.

Last Updated on Friday, 29 June 2012 22:22
 

Log file compression

E-mail Print PDF

Here's something which has bothered me for a long time: I knew that xz is capable of very good compression. I knew that bzip2 is better and slower than gzip. But I never knew how much. So I decided to take one single application and test it. My use case is compressing log files of a server, in this case Java application server logs. The system spits out up to 6GB per day, so good compression is very important.

For the test, I took one of the smaller files, containing 1094 MiB of pure text data, lots of repetitions, lots of numbers.

And I ran a couple of tests - compressing it with bzip2, gzip and xz in all available presets.

Here are some observations beforehand:

  • gzip is always faster than any other compressor
  • gzip -9 compresses better than bzip2 -1 but worse than bzip2 -2
  • bzip2 is very slow compared to the compression ratio achieved
  • xz has a distinct drop in performance between level 3 and 4, also level 4 compresses worse than level 3 - explanation later

Now keep in mind that this is a special use case - a big plain ASCII text file. It's not an academic one though, since log files are quite relevant in many applications.


Now let's look at some graphs, here's compression ratio and time, sorted by ratio ascending.

Compression time and ratio, sorted by ratio

We notice some patterns here:

  • gzip and bzip2's execution times and compression ratios are coherent with the compression level: higher level equals better compression and slower execution
  • same goes for xz level 0 to 3 and xz level 4 to 9

Here an overview of the maximum and minimum times and ratios:

compressor min ratio max ratio min time max time
gzip 7.01% 10.67% 14.4s 31.4s
bzip2 4.85% 8.01% 295s 592s
xz 0-3 4.68% 5.52% 48s 133s
xz 4-9 4.54% 5.42% 193s 755s

Ok, so why did I separate the xz levels like this. It's obvious if you take a look at this chart:

level dict_size mode mf nice depth level mode mf nice depth
0 262144 fast hc3 128 4 0e normal bt4 273 512
1 1048576 fast hc4 128 8 1e normal bt4 273 512
2 2097152 fast hc4 273 24 2e normal bt4 273 512
3 4194304 fast hc4 273 48 3e normal bt4 192 0
4 4194304 normal bt4 16 0 4e normal bt4 273 512
5 8388608 normal bt4 32 0 5e normal bt4 192 0
6 8388608 normal bt4 64 0 6e normal bt4 273 512
7 16777216 normal bt4 64 0 7e normal bt4 273 512
8 33554432 normal bt4 64 0 8e normal bt4 273 512
9 67108864 normal bt4 64 0 9e normal bt4 273 512

Now, the extreme modes aside, the notable difference is that between 3 and 4, mode changes from "fast" to "normal" and the matchfinder algorithm is changes from hash chains to binary tree; the man page also states that "Hash Chains are usually used together with mode=fast and Binary Trees with mode=normal." So we can conclude that, obviously, the fast mode is really fast. The question is, are hash chains generally better for text compression or is the result of the decreased compression ratio from level 3 to 4 a side effect of the much decreased "nice" value or the depth value (depth 0 means the decoder determines a reasonable depth from mf and nice)? I don't have a definite answer to that question, but I know that with identical settings, bt4 compresses the log file in question slower (by a factor ranging from 2 to more than 5) than hc4, and achieves roughly the same compression ratio.


Here's another presentation of the same data:

Compression time and ratio, sorted by time

Again, it's obvious that gz is the fastest, although level 9 is almost indistinguishable from level 8. There is a significant improvement in compression levels for the xz 0-3 levels, and the processing time increases almost linearily upwards, whereas compression ratios vary greatly. Again, all the more evidence to avoid bzip2 for this kind of files.


Finally, I wanted a "tangible" value for the "goodness" of a certain compressor mode. Since compression will always be a tradeoff between time and size, the value of a compressor can be expressed in terms of these two factors. The lower each is, the better, but you need to weigh them. I decided to use this formula:

value = sqrt(time in seconds) * (compression ratio)^2

Compression ratio and rating, sorted by rating

To my surprise, xz -0 wins, the lower xz levels are followed by gz, with bzip being the worst of the bunch. I would have expected the value to increase towards xz -3, that bzip2 is at the low end of the spectrum was to be expected.


Moral of the story

You have large ASCII text files to compress
yes no
how much CPU power do you want to spend on it either you don't have a problem or you need to do similar research
very little it depends I don't care
use gzip use xz -0 to xz -3 or manually fine-tuned use xz -9 or manually fine-tuned

Fine-tuning xz

There is something to be said for fine-tuning the parameters of xz. Although it seems overwhelming at first, it's certainly worth to play with them a little bit if you want to fine-tune your compression. Here's an overview:

parameter values explanation impact on speed impact on compression
mode fast, normal matchfinder mode, fast is only really fast with mf=hc normal is factor 2 slower with mf=bt and factor >5 slower with mf=hc in this test, minimal
mf hc3, hc4, bt2, bt3, bt4 matchfinder - hash chain or binary tree bt is much slower than hc in this test, with identical other parameters, bt fared better than hc, but by a very small margin
dict 4k - 1.5G dictionary size, larger is better the larger the slower, also RAM usage increases the larger the better, although it doesn't make sense to make it larger than the input file size
nice 2-273 length of a "nice" match with mode=fast, higher values are somewhat slower, with mode=normal higher values are a lot slower the higher, the better, but no huge gains
depth 4-100 for mf=hc and 16-1000 fro mf=bt maximum search depth for the match finder CPU impact slightly lower than dict, but defnitely noticable - this is a good second parameter to balance speed vs. compression higher values improve compression, even more so with large input files and large dict size
pb 0-4 position bits, for text files this should be set to 0 almost none pb=0 gives slightly better compression for this use case

Using this knowledge, I was able to compress the tested file down to 36.4MiB (down from 49.7MiB at xz -9, total compression ratio is 1:30 vs. 1:22 at xz -9), although it took almost 51 minutes and a lot of RAM, using this command line:

xz --lzma2=mode=normal,mf=hc4,nice=273,depth=960,dict=512M,pb=0

A more balanced approuch would be

xz --lzma2=mode=fast,mf=hc4,nice=273,depth=64,dict=512M,pb=0

which compressed it to 47.7MiB in just 4:14 minutes (which is still faster and much smaller than any of the bzip2 compression levels, as well as faster and smaller than xz levels 5 to 9), but of course with the huge 512MiB dictionary, the memory usage is also enormous.

 


All tests were performed on an AMD Athlon 64 X2 4600+ (2.4GHz) with 8GB of RAM, so all the tests ran almost entirely in RAM; and the xz version used was 5.0.0 from Debian Squeeze.

 

Last Updated on Thursday, 21 June 2012 22:59
 

Android-x86 progress

E-mail Print PDF

So, it's been a while since I last reported something, so I will now allow people to keep track of the progress on Android-x86 on the IBM Thinkpad X41 Tablet.

Update 7th March 2012: Updated build with new Google Docs, added b43 kernel module and firmware for certain Broadcom WiFi chips, and added radeon (rv100) mesa driver, but I don't think it will work. Build is untested!

New links:


Update 5th March 2012: First try of a "universal" build for all Thinkpads (and likely many more similar devices).

This new image should not only work for the X41, but also on the X61 and maybe the x220 (WiFi doesn't seem to work on this). Changes which I can remember:

  • hdaps module from tp_smapi, so auto-rotation and tilt sensors games could work on the X61 now
  • contains drivers for most components, so if your wifi/graphics/bluetooth/whatever doesn't work it's likely that it will not be so easy to get it to work (missed b43 driver though, will update soon)
  • Pinyin IME for chinese text input
  • battery status updated continuosly
  • internal changes
  • Chih-Wei Huang added the option to never go to sleep (Settings->Display)

New links:


Update 25th Feb 2012: New build out now! Here are the changes (at least those I can remember):

 

  • installer should work again, I probably unintentionally broke it last time. If it still doesn't work for you, follow these steps:
    • when you boot from CD/USB and the selection is shown where you select "Installation"
    • select "Installation" with cursor keys
    • press Tab key
    • press Space key and then type vga=788
    • press Enter key
  • slightly improved performance
  • tablet mode is now detected:
    • software keyboard is shown/hidden accordingly
    • screen rotation is fixed to landscape in "laptop mode" (unless an application explicitely requests portrait)
  • suspend/resume very stable now, I advise to disable screen lock for easier access
  • suspend works also with SD card inserted, but you must not remove or change the SD card while the laptop is sleeping
  • screen bezel buttons now work as described
  • Intel WiFi cards work better now, but if no networks are detected, try:
    • switching WiFi off and on again
    • or add a Wireless Network - it doesn't have to exists, just add a useless network and it should detect existing networks
  • Youtube now works completely (except HD videos), also some Live Wallpapers work
  • added "CatLog" application for easier sending of log files

New links:


This is the old status:

 

  • Working:
    • Android 4.0.3 Ice Cream Sandwich (ICS)
    • WiFi (tested with Intel 2200bg and Atheros 9k mini-PCI cards)
    • Bluetooth (only enabling/disabled and discovery of other devices tested)
    • Audio (make sure the volume is switched on using the keyboard buttons)
    • Touchscreen/Pen input
    • Screen Brightness
    • Orientation Sensor (calibrated at startup, so laptop need to be level when booting)
    • Screen bezel buttons:
      • up/down = Android volume up down
      • Enter = Enter (doesn't work everywhere)
      • ESC = Android "back" key
      • "suitcase" key = Android "menu" key
      • "rotation" key = Android "home" key
      • "hidden" key = Power, to activate standby mode
      • Power key = normal power key behaviour: long-press shows "switch tablet off" dialog
    • Hardware Acceleration (may need to set "Force GPU rendering" in Settings -> Developer options
    • SD card slot
      • when using the SD card slot, sometimes suspend doesn't work properly
      • to use the SD card slot, you need to add SDCARD=/dev/mmcblk0 to the kernel command line (in Grub)
  • Not working/Planned:
    • support for 3G modems (USB dongles/PCMCIA cards)
    • debugging of SD card issues
    • allow for "SD card emulation", real SD card and USB storage at the same time
    • debugging of Intel 2200 WiFi issues

So, if you want to test it out, download it via Torrent:

You need a DHT/trackerless torrents compatible torrent client like Deluge or µTorrent.

For comments or questions, use the Google Groups discussion.

Here is a little video showing the basic features:

Last Updated on Wednesday, 07 March 2012 09:55
 

Debian Squeeze, mytop and Mysql Server 5.1

E-mail Print PDF

After having updated most servers to Debian Squeeze (6.0), I noticed that the packaged mytop utility didn't work with the new mysql-server-5.1. After looking for a solution for a long time, I found out that the variable name for the number of queries has changed in 5.1 - from "Questions" to "Queries" (or maybe the other way round). Thus, a dirty solution is to apply the following patch to /usr/bin/mytop:

--- /usr/bin/mytop	2009-04-06 23:49:09.000000000 +0200
+++ mytop	2011-08-18 10:54:09.000000000 +0200
@@ -794,6 +794,11 @@
         {
             my $key = $ref->{Variable_name};
             my $val = $ref->{Value};
+	    if ($key eq 'Queries') {
+		$key = 'Questions';
+	    } else { if ($key eq 'Questions') {
+		$key = 'Queries';
+	    } }
 
             $STATUS{$key} = $val;
         }

This solved it for me. Of course, the correct solution would be to check the MySQL version and act accordingly, but I leave that as an exercise to the reader :)

Last Updated on Friday, 22 June 2012 20:49
 
  • «
  •  Start 
  •  Prev 
  •  1 
  •  2 
  •  Next 
  •  End 
  • »


Page 1 of 2

All Articles

Home