debian

Scrapy on Debian 6

/ chirale / Leave a comment

Debian 6 comes with Scrapy 0.8 as downloadable packages on apt. Here a quick howto to get this spider works on Debian 6.

  1. sudo apt-get install python-scrapy
  2. cd
  3. mkdir mydir
  4. cd mydir
  5. scrapy-ctl startproject anime
  6. export SCRAPY_SETTINGS_MODULE=anime.settings
  7. export PYTHONPATH=/home/YOURHOMEHERE/mydir

If you’ve already a bot but you, to run your spider thanks to point 6 and 7 you can simply type:

scrapy-ctl crawl example.com

Otherwise, now you can follow the howto on tutorial section of Scrapy 0.8 or this awesome howto by Pravin Paratey to write your own bot, but remember to use the scrapy-ctl command instead of the .py version and to add all your spiders to SCRAPY_SETTINGS_MODULE and PYTHONPATH.

To list your available (and correctly configured) spider, just type:

scrapy-ctl list

If a bot doesn’t appear here, you have an issue on point 6 or 7 or you have a misconfigured spider, i.e. I was forgetting the SPIDER part on bottom of my spider and I was using domain instead of domain_name on my script, see Pravin’s howto to write correct Scrapy 8.0 code.

Raspberry pi as network torrent downloader with Transmission

/ chirale / 27 Comments

I wrote an article about transforming Raspberry Pi into a NAS to share files on a local network with an external usb storage: see technical details there, for brevity here just suppose you’ve a Raspberry PI  on local network with a USB drive attached.

Since Raspberry PI has limited resources, my choice go to Transmission daemon. It’s a very light version of Transmission running as a service (daemon) in background. There are different client available, but I use the Web interface:

Web client

Using web client, I can start and stop and download torrents within the local network and, if you expose raspberry in the DMZ and use a static IP or a dyndns, from everywhere you are.

Let’s install! On root console, type:

# apt-get install transmission-daemon

Edit the configuration file according to your filesystem:

# nano /var/lib/transmission-daemon/info/settings.json

Change “download-dir” and “incomplete-dir” into different directories on your external usb device, something like /media/MYDRIVE/mydownloaddir and /media/MYDRIVE/myincompletedir.

I also changed:

  • “rpc-url”: “/mysecretpath/”
  • “rpc-username”: “mysecretuser”
  • “rpc-password”: “mypassword” note: this will be hashed at the transmission restart
  • “rpc-whitelist”: “127.0.0.1,192.168.*.*,*.*.*.*” warning: copy the *.*.*.* only if you want to expose your Raspberry PI publicly on the Internet.

I leave unchanged:

  • “rpc-port”: 9091

Before restarting transmission, copy the configuration file somewhere as a backup copy, because sometimes Transmission overwrite it badly.

Restart transmission:

# service transmission-daemon restart

Now you should reach web interface in this way:

http://RASPBERRY_PI_LOCAL_IP:rpc-port/rpc-url/

e.g.

http://192.168.0.5:9091/mysecretpath/

or using the host name, for raspbian “raspberrypi”:

http://raspberrypi:9091/mysecretpath/

Typing username and password you can start to add torrent using the first icon on the left (Open torrent):

upload

Uploading a file from your local filesystem or providing a torrent url from an external service.

Important note: please remember that Raspberry PI has limited resources. Do not start too many torrents at once because hash calculations are cpu intensive.

After you finished download, seed for a while and then clear the completed items (right click to show options) you’ve already downloaded to keep Transmission running smoothly.

See also

Turn Raspberry into a small NAS with samba

/ chirale / 13 Comments

I got a Raspberry Pi Model B. It’s cheap and I want to do some experiments for fun.

Experiment #1: I have a 1T external HDD (FAT) and I want to turn Raspberry into a very basic NAS.

I used:

  • 1 External USB HDD (with external power supply)
  • 1 ethernet cable CAT. 5 (10/100) or better
  • 1 HDMI cable and monitor / tv
  • 1 smartphone microusb battery charger
  • 1 SDHC (for the OS)
  • Raspbian “wheezy” (tested on 2012-08-16 release)
  • 1 modem router for connectivity (4 port)
  • 1 Windows PC plugged to the router

I flashed Raspbian into a class 10 SDHC, I follow this useful howto about to turn on HDMI instead of TV and voilà, I got a down-scaled debian system into a silent, little board that I charge with the smartphone charger via microusb (5V, 700mA).

I plug a wireless mouse and keyboard on the first USB port, and then I plug my external drive on the second. Debian read the FAT partition well (mounted on /media/MYDRIVE), but now I have to turn it into a wannabe-NAS.

Shall we dance? With Samba!

I plug the RJ-45 ethernet connector from my modem router into the Raspberry Pi and I follow this howto in Italian.

$ is a pi console (Start > Accessories > LXTerminal)
# is a root console (Start > Accessories > Root terminal)

# adduser guest --home=/home/public --shell=/bin/false --disabled-password
# sudo chmod -R 0700 /home/public
# chown -R guest.guest /home/public
$ sudo apt-get install samba smbfs

Then I have a new user “guest” with no password authentication. The howto covers the creation of a shared home (/home/public) but I do something slightly different (WORKGROUP is my local network name):

editing /etc/samba/smb.conf

## Browsing/Identification ###
# Change this to the workgroup/NT-domain name your Samba server will part of
   workgroup = WORKGROUP

####### Authentication #######
   security = share

   obey pam restrictions = yes
   guest account = guest
   invalid users = root

And now the most interesting part:

[MYDRIVE]
comment = Mydrive
read only = no
locking = no
path = /media/MYDRIVE
guest ok = yes
force user = pi

Where /media/MYDRIVE is the path to your external usb drive.

And then:

# /etc/init.d/samba restart

to apply.

As this howto explains, the “force user” allows a user (i.e. guest) to get the files from a device mounted by another user (i.e. pi, the default raspbian user).

Have fun

Now on the Windows machine on the Network panel I look for RASPBERRYPI and inside it I find the “mydrive” folder, with all the files from MYDRIVE within. I play a 720p video without slowdown. And so, the cheap NAS experiment is successfully completed.

Howto extract tracks from mkv and avi

/ chirale / 4 Comments

This howto requires:

  • mplayer
  • mkvtoolnix
  • your Linux box 😉

Audio from Avi files (es. Xvid + MP3):

mplayer -dumpaudio "mymovie.avi" -dumpfile mymovie_audio_track.mp3

Tracks from Matroska MKV file:

List all tracks:

mkvmerge -i mymovie.mkv

File 'mymovie.mkv': container: Matroska
Track ID 1: video (V_MS/VFW/FOURCC, XVID)
Track ID 2: audio (A_VORBIS)
Track ID 3: audio (A_VORBIS)
Track ID 4: subtitles (S_TEXT/UTF8)
Track ID 5: subtitles (S_TEXT/UTF8)

mkvextract tracks *.mkv 3:mymovie_audio_track.ogg 4:mymovie_subtitle.srt

Creates two files, mymovie_audio_track.ogg (track 3) and mymovie_subtitle.srt (track 4).

Installing Plone on Debian

/ chirale / Leave a comment

A little howto to quickly install and try Plone (a GPL’d CMS based on Zope) on your linux box. Well, the installer seems to do the job nicely. 🙂

Tested on Plone 3.* version, Debian “Lenny”.

    • apt-get install g++
    • Download latest version of Plone (Unified Installer)
    • Execute:
      tar zxvf Plone-YOURVERSION-UnifiedInstaller.tgz
      cd Plone-      YOURVERSION-UnifiedInstaller
      ./install.sh standalone
      gedit /usr/local/Plone/zinstance/README.txt &
      gedit /usr/local/Plone/zinstance/buildout.cfg &
      /usr/local/Plone/zinstance/bin/plonectl start
      less /usr/local/Plone/zinstance/adminPassword.txt

README should be read to follow installation instructions, then you can modify Plone configuration on buildout.cfg, and then you can start Plone. On adminPassword.txt you’ll find your Plone passwords to use for administrative purpouses.

  • Add /usr/local/Plone/zinstance/bin/plonectl start to /etc/rc.local before exit 0 (Red Hat) to run plone at any server restart or create a script on /etc/init.d/ (Debian) like.

Now you can test this CMS based on Python (I’ve tested it 4 years ago, maybe it hardly can replace Drupal but you can give it a try 😉 ).

Mount NTFS partition on Debian

/ chirale / Leave a comment

Once upon a time, writing to NTFS was a chimera. For years I tell to friends to use FAT32 when installing Windows XP. But now the NTFS support is fine, even on Debian. Just type on your shell:

sudo apt-get install ntfs-3g

sudo mkdir /media/win

And add to your /etc/fstab:

/dev/YOURDEVICE /media/win ntfs-3g umask=0,nls=utf8 0 0

Where YOURDEVICE is /dev/something you can found on Partition Editor for the your NTFS partition. Do a mount -a or reboot to apply changes, and then go to /media/win directory to start to read.

See also:

PEAR install quick howto on Debian Lenny: Image_Graph

/ chirale / 4 Comments

Tested on Debian Lenny.

Time elapsed before a working script < 10 minutes 😉

Install PEAR installer:

# apt-get install php-pear

First, type:

pear -h

to get help.

On root shell type:

# pear install --alldeps Image_Graph

Install failed with error:

Failed to download pear/Image_Graph within preferred state “stable”, latest release is version 0.7.2, stability “alpha”, use “channel://pear.php.net/Image_Graph-0.7.2” to install

Retype as:

# pear install --alldeps Image_Graph-0.7.2

Repeat for other alpha dependencies the same procedure, eg:

pear install --alldeps Numbers_Words-0.15.0

At last, you got:

Nothing to install

Now, you can find all installed packages on /usr/share/php.

ImageGraph is installed on /usr/share/php/Image/Graph.php.

You have to include it on your PHPs.

You can follow the Ian’s howto on:

http://www.phpbuilder.com/columns/ian_gilfillan20060503.php3

To start using PEAR Image_Graph.

Here the result using Ian’s code on my Linux box:

Image_Graph test, using Ian code

Image_Graph test, using Ian code

Additional info:

Hosting providers (eg. HostMonster) sometimes furnish simplified PEAR installation via CPanel. So, you can use local PEAR installation for developement, and online installation for production use.

Remember:

  • To check differences between local and production versions
  • To change include path! (say no to white screen!)

See also:

Change framebuffer resolution

/ chirale / 2 Comments

While trying to update kernel in order to run Virtualbox, i got a weird resolution error. On OS boot, screen blanks with an error like “Resolution is beyond allowed limits”. Since this happens before GUI is loaded, Xorg is innocent. I’ve installed the 915resolution patch for a widescreen HP, but it’s innocent too.

Well, this is a framebuffer issue! To solve it I’ve followed these instructions on technobounce.

  • apt-get install hwinfo
  • hwinfo –framebuffer
  • Then make a backup copy of your /boot/grub/menu.lst and add to your kernel line vga=0x035a where 0x035a is the wanted resolution mode.

Netgear WG111 on Debian Lenny with ndiswrapper

/ chirale / 5 Comments

Tested on: Debian Etch, Lenny/Sid (Testing), kernel 2.6

  1. Plug WG111 and type lsusb to verify hardware is plugged. You should get something like:
    Bus 003 Device 002: ID 0000:0000 NetGear, Inc. WG111 WiFi (v2)
    Bus 003 Device 001: ID 0000:0000
    Bus 001 Device 001: ID 0000:0000
    Bus 002 Device 001: ID 0000:0000
  2. On root shell, type:
    apt-get install module-assistant
    m-a prepare
    m-a a-i ndiswrapper
    modprobe ndiswrapper
    Now ndiswrapper module is loaded. (if you got a FATAL: Module ndiswrapper not found fatal error, repeat these steps)
  3. If interface is successfully modprobed, then type
    update-modules
    To create ndiswrapper module config files.
  4. Add
    ndiswrapper
    to /etc/modules to load this module on boot.

  5. apt-get install wlassistant ndisgtk ndiswrapper-utils
    Install some utilities to configure wireless interface.
  6. Run ndisgtk (or type ndiswrapper -i /media/cdrom0/ndis5/netwg111.inf, where cdrom0 is Netgear driver disc)
  7. Select Netgear driver from CD-ROM (netwg111.inf): it’ll be copied automatically to /etc/ndiswrapper
  8. Copy other files under /media/cdrom0/ndis5 in /etc/ndiswrapper/netwg111/
  9. If interface is not detected, type ndiswrapper -m to write modprobe config files for it.
  10. Use gksu network-admin or gksu wlassistant to configure your wireless connection.

Tip: If after system upgrading you cannot access to wireless network, repeat step (2).

See also: