debian

Scrapy on Debian 6

/ chirale / Leave a comment

Debian 6 comes with Scrapy 0.8 as downloadable packages on apt. Here a quick howto to get this spider works on Debian 6.

  1. sudo apt-get install python-scrapy
  2. cd
  3. mkdir mydir
  4. cd mydir
  5. scrapy-ctl startproject anime
  6. export SCRAPY_SETTINGS_MODULE=anime.settings
  7. export PYTHONPATH=/home/YOURHOMEHERE/mydir

If you’ve already a bot but you, to run your spider thanks to point 6 and 7 you can simply type:

scrapy-ctl crawl example.com

Otherwise, now you can follow the howto on tutorial section of Scrapy 0.8 or this awesome howto by Pravin Paratey to write your own bot, but remember to use the scrapy-ctl command instead of the .py version and to add all your spiders to SCRAPY_SETTINGS_MODULE and PYTHONPATH.

To list your available (and correctly configured) spider, just type:

scrapy-ctl list

If a bot doesn’t appear here, you have an issue on point 6 or 7 or you have a misconfigured spider, i.e. I was forgetting the SPIDER part on bottom of my spider and I was using domain instead of domain_name on my script, see Pravin’s howto to write correct Scrapy 8.0 code.

Advertisements

Raspberry pi as network torrent downloader with Transmission

/ chirale / 27 Comments

I wrote an article about transforming Raspberry Pi into a NAS to share files on a local network with an external usb storage: see technical details there, for brevity here just suppose you’ve a Raspberry PI  on local network with a USB drive attached.

Since Raspberry PI has limited resources, my choice go to Transmission daemon. It’s a very light version of Transmission running as a service (daemon) in background. There are different client available, but I use the Web interface:

Web client

Using web client, I can start and stop and download torrents within the local network and, if you expose raspberry in the DMZ and use a static IP or a dyndns, from everywhere you are.

Let’s install! On root console, type:

# apt-get install transmission-daemon

Edit the configuration file according to your filesystem:

# nano /var/lib/transmission-daemon/info/settings.json

Change “download-dir” and “incomplete-dir” into different directories on your external usb device, something like /media/MYDRIVE/mydownloaddir and /media/MYDRIVE/myincompletedir.

I also changed:

  • “rpc-url”: “/mysecretpath/”
  • “rpc-username”: “mysecretuser”
  • “rpc-password”: “mypassword” note: this will be hashed at the transmission restart
  • “rpc-whitelist”: “127.0.0.1,192.168.*.*,*.*.*.*” warning: copy the *.*.*.* only if you want to expose your Raspberry PI publicly on the Internet.

I leave unchanged:

  • “rpc-port”: 9091

Before restarting transmission, copy the configuration file somewhere as a backup copy, because sometimes Transmission overwrite it badly.

Restart transmission:

# service transmission-daemon restart

Now you should reach web interface in this way:

http://RASPBERRY_PI_LOCAL_IP:rpc-port/rpc-url/

e.g.

http://192.168.0.5:9091/mysecretpath/

or using the host name, for raspbian “raspberrypi”:

http://raspberrypi:9091/mysecretpath/

Typing username and password you can start to add torrent using the first icon on the left (Open torrent):

upload

Uploading a file from your local filesystem or providing a torrent url from an external service.

Important note: please remember that Raspberry PI has limited resources. Do not start too many torrents at once because hash calculations are cpu intensive.

After you finished download, seed for a while and then clear the completed items (right click to show options) you’ve already downloaded to keep Transmission running smoothly.

See also

Turn Raspberry into a small NAS with samba

/ chirale / 13 Comments

I got a Raspberry Pi Model B. It’s cheap and I want to do some experiments for fun.

Experiment #1: I have a 1T external HDD (FAT) and I want to turn Raspberry into a very basic NAS.

I used:

  • 1 External USB HDD (with external power supply)
  • 1 ethernet cable CAT. 5 (10/100) or better
  • 1 HDMI cable and monitor / tv
  • 1 smartphone microusb battery charger
  • 1 SDHC (for the OS)
  • Raspbian “wheezy” (tested on 2012-08-16 release)
  • 1 modem router for connectivity (4 port)
  • 1 Windows PC plugged to the router

I flashed Raspbian into a class 10 SDHC, I follow this useful howto about to turn on HDMI instead of TV and voilà, I got a down-scaled debian system into a silent, little board that I charge with the smartphone charger via microusb (5V, 700mA).

I plug a wireless mouse and keyboard on the first USB port, and then I plug my external drive on the second. Debian read the FAT partition well (mounted on /media/MYDRIVE), but now I have to turn it into a wannabe-NAS.

Shall we dance? With Samba!

I plug the RJ-45 ethernet connector from my modem router into the Raspberry Pi and I follow this howto in Italian.

$ is a pi console (Start > Accessories > LXTerminal)
# is a root console (Start > Accessories > Root terminal)

# adduser guest --home=/home/public --shell=/bin/false --disabled-password
# sudo chmod -R 0700 /home/public
# chown -R guest.guest /home/public
$ sudo apt-get install samba smbfs

Then I have a new user “guest” with no password authentication. The howto covers the creation of a shared home (/home/public) but I do something slightly different (WORKGROUP is my local network name):

editing /etc/samba/smb.conf

## Browsing/Identification ###
# Change this to the workgroup/NT-domain name your Samba server will part of
   workgroup = WORKGROUP

####### Authentication #######
   security = share

   obey pam restrictions = yes
   guest account = guest
   invalid users = root

And now the most interesting part:

[MYDRIVE]
comment = Mydrive
read only = no
locking = no
path = /media/MYDRIVE
guest ok = yes
force user = pi

Where /media/MYDRIVE is the path to your external usb drive.

And then:

# /etc/init.d/samba restart

to apply.

As this howto explains, the “force user” allows a user (i.e. guest) to get the files from a device mounted by another user (i.e. pi, the default raspbian user).

Have fun

Now on the Windows machine on the Network panel I look for RASPBERRYPI and inside it I find the “mydrive” folder, with all the files from MYDRIVE within. I play a 720p video without slowdown. And so, the cheap NAS experiment is successfully completed.

Howto extract tracks from mkv and avi

/ chirale / 4 Comments

This howto requires:

  • mplayer
  • mkvtoolnix
  • your Linux box 😉

Audio from Avi files (es. Xvid + MP3):

mplayer -dumpaudio "mymovie.avi" -dumpfile mymovie_audio_track.mp3

Tracks from Matroska MKV file:

List all tracks:

mkvmerge -i mymovie.mkv

File 'mymovie.mkv': container: Matroska
Track ID 1: video (V_MS/VFW/FOURCC, XVID)
Track ID 2: audio (A_VORBIS)
Track ID 3: audio (A_VORBIS)
Track ID 4: subtitles (S_TEXT/UTF8)
Track ID 5: subtitles (S_TEXT/UTF8)

mkvextract tracks *.mkv 3:mymovie_audio_track.ogg 4:mymovie_subtitle.srt

Creates two files, mymovie_audio_track.ogg (track 3) and mymovie_subtitle.srt (track 4).

Installing Plone on Debian

/ chirale / Leave a comment

A little howto to quickly install and try Plone (a GPL’d CMS based on Zope) on your linux box. Well, the installer seems to do the job nicely. 🙂

Tested on Plone 3.* version, Debian “Lenny”.

    • apt-get install g++
    • Download latest version of Plone (Unified Installer)
    • Execute:
      tar zxvf Plone-YOURVERSION-UnifiedInstaller.tgz
      cd Plone-      YOURVERSION-UnifiedInstaller
      ./install.sh standalone
      gedit /usr/local/Plone/zinstance/README.txt &
      gedit /usr/local/Plone/zinstance/buildout.cfg &
      /usr/local/Plone/zinstance/bin/plonectl start
      less /usr/local/Plone/zinstance/adminPassword.txt

README should be read to follow installation instructions, then you can modify Plone configuration on buildout.cfg, and then you can start Plone. On adminPassword.txt you’ll find your Plone passwords to use for administrative purpouses.

  • Add /usr/local/Plone/zinstance/bin/plonectl start to /etc/rc.local before exit 0 (Red Hat) to run plone at any server restart or create a script on /etc/init.d/ (Debian) like.

Now you can test this CMS based on Python (I’ve tested it 4 years ago, maybe it hardly can replace Drupal but you can give it a try 😉 ).

Mount NTFS partition on Debian

/ chirale / Leave a comment

Once upon a time, writing to NTFS was a chimera. For years I tell to friends to use FAT32 when installing Windows XP. But now the NTFS support is fine, even on Debian. Just type on your shell:

sudo apt-get install ntfs-3g

sudo mkdir /media/win

And add to your /etc/fstab:

/dev/YOURDEVICE /media/win ntfs-3g umask=0,nls=utf8 0 0

Where YOURDEVICE is /dev/something you can found on Partition Editor for the your NTFS partition. Do a mount -a or reboot to apply changes, and then go to /media/win directory to start to read.

See also: