Upgrade from Windows 7 OEM to Windows 10

Can still be feasible to upgrade an existing Window 7, 32 bit installation on 2020? I tried and yes, it is.

Actually this is a suggestion of some of the best articles on this topic, this ZDnet howto by Ed Bott (archive) and this howtogeek howto by Chris Hoffman (archive) to convert a Windows 10 32bit to 64bit.

Tested upgrade path

I’ve walked a long installation path, starting from an 11 years old DVD. This is not the ideal scenario, but I got a very clean state to start from.

  1. Install Windows 7 Professional OEM from DVD or restore from system partition to get a clean state
  2. Upgrade Windows 7 Professional using Windows Update
  3. Upgrade to Windows 10 Pro 32bit, preserving the reserved partitions using the Upgrade this PC now path
  4. From Windows 10 32 bit, create an installation USB with Windows 10 Media creation tool choosing to create a media for an Other PC
  5. Install the new system above the old, preserving the reserved partitions

Tips

The howtos cited explains the above steps, but here are some supplmental tips on different scenarios, plus some things that maybe aren’t so clear at start. I’ve tested all of these:

  • You need an installed Windows 7 version to do the update following the howtos above.
  • Do not mess with reserved partitions: I don’t know if it will broke the upgrade path, but I don’t and it works.
  • You can use Clonezilla to make copies of your OS any time. Personally I do:
    • Before the Windows 7 to Windows 10 update (between 2 and 3)
    • After the 32bit installation (before 4)
    • After the 64bit installation (after 5)
  • Using Clonezilla, you can easily change the Hard Drive keeping the previous state of the OS intact if something goes wrong, ready to be restored.
  • If you have a Windows 7 Professional OEM, you will get a Windows 10 Pro with auto-activated digital license.
  • If you have Windows 7 on an unreadable DVD, try to change the reader before trying to clean up the surface.
  • If you have a DVD, make a copy with something like k3b and flash it on an USB with WoeUSB or similar to speed up the installation

 

Migrate mercurial code hosting from Bitbucket to your server in 9 steps using docker

By now Atlassian is dropping support to Mercurial on the popular Bitbucket service. Here is a proof of concept to use a Docker container as a separate environment where self-host your code using basic mercurial features without bells and whistles.

To do so, a docker container based on popular and lightweight jdeathe/centos-ssh image will be used. In this example, it’s supposed to use a remote server with docker service up and running.

1. Generate public / private pair

Create new keys to authenticate to the new container. Protect it with a password to deploy on external servers safely. In this example, an EdDSA type key is used.

ssh-keygen -t ed25519 -C "Key for xxx at xxx on xxx"

2. Choose keys and passwords

Choose a name for your new container here:

export SSHCONTAINER=mycodehosting.example.org

Create a new file named .env with the following content plus a custom:

  • content of the generated .pub file in the AUTHORIZED_KEYS row
  • a strong password to switch from hg to root via sudo su – root
  • timezone
SSH_AUTHORIZED_KEYS=*******PASTE PUB KEY here ***********
SSH_CHROOT_DIRECTORY=%h
SSH_INHERIT_ENVIRONMENT=false
SSH_PASSWORD_AUTHENTICATION=false
SSH_SUDO=ALL=(ALL) ALL
SSH_USER=hg
SSH_USER_FORCE_SFTP=false
SSH_USER_HOME=/home/%u
SSH_USER_ID=500:500
SSH_USER_PASSWORD=*******STRONG PASSWORD HERE (without ")***********
SSH_USER_PASSWORD_HASHED=false
SSH_USER_PRIVATE_KEY=
SSH_USER_SHELL=/bin/bash
SYSTEM_TIMEZONE=********YOUR TIMEZONE HERE e.g. Europe/Rome***********

This configuration:

  • Allow the connection using the private key generated before
  • Disable password authentication
  • Set default user name to hg
  • Allow all users to switch to sudo (there will be only an hg user)
  • Set server with preferred timezone

3. Create the centos-ssh container

On the same directory where resides the .env file before, create a new container:

docker run -d \
  --name $SSHCONTAINER \
  -p 12120:22 \
  --env-file .env \
  -v /opt/path/to/some/host/dir:/home \
  jdeathe/centos-ssh
  • create a detached container named $SSHCONTAINER
  • expose the container on port 12120. If you want lo limit to localhost only, use 127.0.0.1:12120:22 or iptables will be set up to bind 0.0.0.0 because docker mess up with iptables. You can also disable iptables on docker.
  • map the whole container /home directory to a new directory created by root on host /opt/path/to/some/host/dir:

Note: do not use ACL (e.g. setfacl) on /opt/path/to/some/host/dir or .ssh directory will broke (e.g. Bad owner or permissions)

4. Install mercurial on container

Now on container install mercurial and its dependencies. You can login as root using docker:

docker exec -it $SSHCONTAINER bash

or saving this script then chmod a+x it and launch:

#!/bin/bash
set -e
docker exec -it -u root $SSHCONTAINER  yum install -y python36-devel python36-setuptools gcc
docker exec -it -u root $SSHCONTAINER /usr/bin/pip3 install mercurial

Restart the container:

docker container restart $SSHCONTAINER

Then check if mercurial is running for user hg:

docker exec -it -u hg $SSHCONTAINER hg --version

Then if container is running smoothly, you can update it to restart always on reboot or on docker service restart:

docker container update $SSHCONTAINER --restart always

then check if it’s applied:

docker container inspect $SSHCONTAINER | grep -B0 -A3 RestartPolicy

5. Login to container directly

Now on your local machine you can connect directly to the container using SSH without caring about the host.

By default an iptables rule is created by docker to allow connections from outside. Anyway, you have to specify the port and the user name .ssh/config like this:

Host mycodehosting.example.org
    Hostname mycodehosting.example.org
    User hg
    Port 12120
    PreferredAuthentications publickey
    IdentityFile /home/chirale/.ssh/id_ed25519_mycodehosting_example_org

This configuration is useful when you create a subdomain exclusively to host code, then you associate it a port and a username to obtain a mercurial url like this:

ssh://hg@mycodehosting.example.org/test/project

where dir and subdir are directly in /home/hg directory of container, on host /opt/path/to/some/host/dir/hg/test/project. Differently from Bitbucket, you can have how many  directory level you want to host the project.

6. Create a test repo

Create a test repository inside this container. You can access everywhere with the above ssh configuration using:

ssh mycodehosting.example.com

Then you can

cd repo/
ls
mkdir alba
cd alba/
hg init
hg verify
checking changesets
checking manifests
crosschecking files in changesets and manifests
checking files
checked 0 changesets with 0 changes to 0 files
cat > README.txt
Read
this
CTRL+D
hg addremove
adding README.txt
hg commit -m "First flight"
abort: no username supplied
(use 'hg config --edit' to set your username)
hg config --edit
hg commit -m "First flight"

7. Clone the test repo

Then from everywhere you can clone the repo adding :

hg clone ssh://mycodehosting.example.org/repo/alba

You can commit and push now.

If you login to mycodehosting.example.org, no new file was added. You’ve simply to run

hg update

to get it. Note that you haven’t to update every time you push new commits on alba to mycodehosting.example.org. Simply all changes are recorded, but not yet reflected on directory structure inside container.

If this is a problem for you, you can automate the update every time hg has a new changeset using for example supervisor service, shipped with centos-ssh.

Compare these:

parent: 1:9f51cd87d912 tip
 Second flight
branch: default
commit: (clean)
update: (current)
hg summary
missing pager command 'less', skipping pager
parent: 1:9f51cd87d912
 Second flight
branch: default
commit: (clean)
update: 1 new changesets (update)

The first hasn’t change update: (current), the second has update: 1 new changesets (update).

8. Migrate the code from Bitbucket to self-host

From the container, logged as hg user, import temporary your key to download the repository from old bitbucket location following Bitbucket docs, then:

cd ~
mkdir typeofproject
cd typeofproject
hg clone ssh://hg@bitbucket.org/yourbbuser/youroldbbrepo

Then you can alter the directory as you like:

  • edit the .hg/hgrc file changing parameters as you like
  • rename youroldbrepo directory

Remember to temporary store on the container the ssh keys and config to access to Bitbucket if any (permission should be 600). You can remove these keys when migration is done.

After a test clone you can drop the Bitbucket repo.

9. Find your flow

With a self-hosted solution you have to maintain the service. This is a relatively simple solution to set up and maintain.

If you are comfortable with old Bitbucket commit web display, you can use PyCharm to see a nice commit tree like this:

Tested on release 2.6.1 with centos:7.6.1810.

 

Help us fight COVID-19 in Piedmont (Archived)

For more than twelve years I write on this blog about IT stuff. I’ve provided a free service in English, even if Italian is my mother tongue, to reach the widest audience. Now this little service is at risk, because my family, friends and I are.

This page is archived and it is not updated anymore

I live in Italy, more precisely in Piedmont. After Lombardy it was one of the first region to close schools and cancel public events before the national lockdown. We cannot leave our homes without a valid reason.

Here there’s a public healthcare system, but handled region by region. At the time I’m writing, Lombardy is about to fill all ICUs. There are conflicts between central government in Rome and regions.

They are not receiving enough materials and support from Rome or it’s too slow, so regions have to handle the emergency with slow responses from central government. Lockdown decided centrally is useful, but there aren’t enough personal protective equipment for the healthcare personnel so nurses and doctors become sick in areas where contagion is already spread. Here’s the real-time status.

I don’t know if Piedmont will be the next facing the same fate, so region needs resources to avoid the collapse of the healthcare system.

Please donate if you can all instruction in this institutional, verified link:

What we’ll do here can be useful to know how to fight the COVID-19 in other countries.

Some practices we are experimenting are described below.

Update 2020-05-15:

Update 2020-04-15:

  • 5 millions of surgical masks will be bought by Regional authorities and will be distributed to population for free using part of donations from the official bank account linked above. Thank you donors!
  • Regional headquartered DiaSorin complete the tests in Italy for new rapid test on March, 9th. The product will be available within a month.

Update 2020-03-30:

Update 2020-03-20:

  • 4 rapid tests devices by DiaSorin are available on two Turin hospitals according to regional authorities. These new devices will process 16 samples every 90 minutes.
  • 7 Piedmont ER was shut down to reassign medical personnel to handle COVID-19 pandemic
  • 4 Arpa (local environment protection agency) laboratories produced 568 liters of hand sanitizer gel to deliver to police and medical personnel
  • Rural tourism operators can now run home delivery services to both address supply of meals from population and try to save these activities after the crash of tourism demand
  • People under quarantine haven’t to waste sorting. All waste should be disposed in two different plastic bag, one inside another, following strict rules. To avoid contamination, waste sorting facilities (isole ecologiche) are closed.

Update 2020-03-17:

  • A new regional hospital for Covid-19 patients will be opened on 2020-03-22, setup is underway 1
  • Regional production of personal protection devices is underway converting production of plants like Miroglio
  • Regional headquartered DiaSorin received US Federal Funds to develop a rapid Covid-19 test to be submitted to FDA by the end of March 2020

 

Give flatpack a chance

Talking about package manager on Linux, flatpak gained attention recently. Installation is easy, this is about installing on Ubuntu.

If you’ve some trouble installing an application using the package manager shipped with your distro, you can give it a try, since it’s available on 22 distro by now.

In the above case, torbrowser packet is broken on a Ubuntu 18.04, as many users noted out on the package page. Installing the package via flatpak, everything run smoothly in minutes.

Simply:

flatpak search PACKAGENAME

Then:

flatpak install PACKAGENAME

Command line output is very complete, but if you want to use a GUI, on Ubuntu gnome-software (Ubuntu Software) will list the flatpak packages too, just read the package info on bottom of the description page and search packages as usual.

Full backup of PostgreSQL and restore on docker

In this howto I will show how to backup all PostgreSQL databases, roles and permission on a server and restore them on another server on a docker.

1. Backup databases on origin server

To backup all PostgreSQL databases automatically, preserving roles and permission, you can create this cron.d script (this example is from a CentOS):

# Dump postgresql database automatically
SHELL=/bin/bash
PATH=/sbin:/bin:/usr/sbin:/usr/bin
MAILTO=root
40 22 * * 0 postgres /bin/bash -c 'pg_dumpall -Upostgres | gzip > "/var/backup/myexport.sql.gz"'

In order:

  1. Run on at 22:40 on Sunday
  2. Run as postgres user (cron is executed by root, and root can impersonate it without password)
  3. Use the bash interpreter (not sh)
  4. Run pg_dumpall to dump all database
  5. Pipe the output to gzip to compress the SQL into a /var/backup/myexport.sql.gz file

2. Transfer on another server

Transfer the dump using ssh using something like this:

rsync -rltvz --no-o --no-g myuser@originserver:/var/backup/dumpall.sql.gz /opt/backup/myproj/data/

Use .ssh/config to store connection info.

3. Restore on a clean docker container

The following bash script will create a docker container, populating it with

set -e
echo "Remove identical container, keep data on directory"
NOW=`date +%Y-%m-%d`
# create an id based on timestamp plus MAC address 
UNIQUID=`uuidgen -t`
NAME="postgresql-myproj-$NOW-$UNIQUID"
echo "Get postgres docker image"
docker pull postgres:9.4-alpine
echo "Generate 1 password with 48 char length"
PPASS=`pwgen 48 1`
mkdir -p /opt/docker_data/$NAME
echo "Save psql password on /opt/docker_data/$NAME.pwd"
echo $PPASS > "/opt/docker_data/$NAME/psql.pwd"
echo "extract database"
mkdir -p /opt/docker_data/$NAME/psqldata
mkdir -p /opt/docker_data/$NAME/share
gunzip -c /opt/backup/myproj/data/dumpall.sql.gz > /opt/docker_data/$NAME/share/restore.out
echo "Run a clean docker"
docker run --name $NAME -e POSTGRES_PASSWORD=$PPASS -d -p 44432:5432 -v /opt/docker_data/$NAME/psqldata:/var/lib/postgresql/data -v /opt/docker_data/$NAME/share:/extshare --restart always postgres:9.4-alpine
sleep 10
echo "Restore from /extshare/restore.out using user postgres (-upostgres) the database postgres (all dbs)"
docker exec -upostgres $NAME psql -f /extshare/restore.out postgres
echo "Clear the restore.out file"
rm /opt/docker_data/$NAME/share/restore.out

In order, this script:

  1. Download a postgres:9.4-alpine image (choose your version)
  2. Generate a random container name postgresql-myproj-YYYY-MM-DD-random-id based on MAC address and timestamp
  3. Generate a random password and save on a file
  4. Generate a directory structure on host system to keep postgres file and dump outside docker
  5. Create a new postgres container exposed to host on port 44432
  6. Save postgres files on /opt/docker_data/$NAME/psqldata on host
  7. Expose the dump file directory on /extshare on guest
  8. Restore the dump using role postgres
  9. Delete the dump

Resulting directory structure on host will be:

.
└── postgresql-myproj-2019-28-12-*******
    ├── psqldata [error opening dir]
    ├── psql.pwd
    └── share
        └── restore.out

Then to connect just use the same user and password of the database on origin server.

Permissions

Usually on /opt/docker_data/$NAME/psqldata/pg_hba.conf, you’ve to add a line like this:

host all all 172.17.0.1/32 md5

giving to host (reachable by 172.17.0.1 inside docker) full access to database. But the default image ship a handy, permissive entry:

host all all all md5

So you can connect without any step to the database.

Connection

If you connect to the destination server with ssh, if you cannot access the port remember to forward the PostgreSQL on .config like:

Host destinationsrv
    Hostname destinationaddr
    User mydestservuser
    IdentityFile /home/myuser/.ssh/id_rsa_destination_server
    LocalForward 44432 127.0.0.1:44432

Remember that if you haven’t a firewall, docker container will start with a permissive rule like:

0.0.0.0:44432->5432/tcp

So will be exposed and reachable using password.

Disable search and autocomplete on url bar of Firefox

The default configuration of Firefox can cause a search to be performed when an url is wrong using the main bar, or auto-suggestion to be performed silently. This setting may disclose to third parties local path when mistyped.

At the end of this howto, Firefox will be back to a status where the user take back more control of searches, reducing disclosed data to external services. You can also disable the suggestions when typing.

How to disable automatic search

To disable search on url bar of Firefox, first re-enable the double search bar:

  1. Go to Settings
  2. Go to Search settings
  3. On Search Bar, select the 2 bar option

Now disable the search on the url bar:

  1. Visit on the main bar about:config
  2. Accept to continue
  3. Search “keyword”
  4. Change the status from true to false (double click on it)

Now if you try to type a keyword, you will be redirected to the keyword.com website, e.g. boh will redirect to boh.com.

Data aren’t sent anymore to other search engines except for autocompletes, but you can disable it following next steps.

Disable search autocomplete on urlbar

  1. Go to about:config
  2. Search browser.urlbar.suggest.searches
  3. Set value from true to false (double click on it)

Now if you type a search on the url bar, the search will be performed on the history alone, but if you use the search bar autocomplete will works normally.

Disable search autocomplete on search bar

To completely disallow search suggestion on the new right searchbar:

  1. Look for browser.search.suggest.enabled on about:config
  2. Double click on it to set false

Disable auto .com

Note: if you wrongly type a search in the url bar, it will be autocompleted as YOUR-SEARCH.com (YOUR SEARCH + “.com”).

To disable the “.com” suffix or the “www” prefix:

  1. Look for browser.fixup.alternate.enabled on about:config
  2. Double click on it to set false

Final behaviour

With all these changes applied:

  1. The url bar will look for a URL exactly as you’ve typed (except for the https / http that will be autocompleted)
  2. The search bar will perform the search on the search engine you’ve specified
  3. No silent request will be forwarded to search engines on url bar or search bar (as far as I know)

Sort of CDN to serve client-side libraries via an auto-pull git repo on tmpfs

This configuration will allow to install on a Debian-based system a fast server for client libraries. Key technologies used are:

  • tmpfs to serve files from volatile memory
  • git / mercurial from github / bitbucket to get files from a public or private repository
  • systemd units to mount tmpfs and sync
  • nginx to serve files to user

On this first step you’ll create a service to reserve some RAM for static files, pulling them from a private or public repo.

Mount tmpfs with systemd

To serve files directly from RAM, you have to mount a tmpfs directory. You can do it on fstab:

/etc/fstab

tmpfs /mnt/cdn tmpfs rw,nodev,nosuid,size=300M 0 0

Or with a systemd unit:

/etc/systemd/system/mnt-cdn.mount

[Unit]
Description=Mount empty CDN directory on volatile memory

[Mount]
What=tmpfs
Where=/mnt/cdn
Type=tmpfs
Options=defaults,noatime,size=300M

[Install]
WantedBy=multi-user.target
  • noatime will disable last access on contained files, reducing write on disk
  • size will reserve 300MB for /mnt/cdn partition on RAM (increase as needed)
  • WantedBy=multi-user.target mount the partition on runlevel 3 (multi-user mode with networking)

Create two units on a local path like /usr/local/share/systemd then create a symlinks on /etc/systemd/system or create directly them on /etc/systemd/system. You can also directly create them on /usr/local/share/systemd.

Create the pull service

When the /mnt/cdn is successfully loaded, pull static files from your repository.

/etc/systemd/system/cdn-pull.service

[Unit]
Description=Pull on CDN directory.
After=network-online.target

[Service]
User=youruserhere
Group=youruserhere
ExecStart=/usr/local/share/systemd/cdn-pull.sh

[Install]
WantedBy=mnt-cdn.mount
  • Clone the git repository with a user on system using a key with an alias
  • Change youruserhere to the user who cloned the repository
  • Add to /root/.ssh/config and to  /root/.ssh/my_private_key the private key to do the pull

Meaning:

  • WantedBy=mnt-cdn.mount copy the files to RAM only after the /mnt/cdn is created
  • After=network-online.target pull the repository only when the network is ready

On pull, all files will be written by root as youruserhere:youruserhere.

After the pull, to reduce RAM occupation, this script doesn’t download directly to RAM .git directory but copy them with rsync excluding them:

/usr/local/share/systemd/cdn-pull.sh

#!/bin/bash
# stop on first error
set -e
cd /srv/cdn-all
git pull
exec rsync -a --exclude=.git --exclude=.gitignore /srv/cdn-all/* /mnt/cdn/

Get systemd to know about the mount and service

To reload systemd units, you have to

systemctl daemon-reload

Then do the mount via the systemd unit:

systemctl start mnt-cdn.mount

Enable on boot

Since the cdn-pull.service is tied to mnt-cdn.mount, both have to be enabled to run:

systemctl enable mnt-cdn.mount
systemctl enable cdn-pull.service
  1. When the system is ready create the tmpfs on /mnt/cdn/
  2. After tmpfs is successfully created by the unit, the file will be automatically synced through cdn-pull.service.

Mount will auto-start sync

Start only the mnt-cdn.mount:

systemctl start mnt-cdn.mount

And then ask for info about both services:

systemctl status mnt-cdn.mount
systemctl status cdn-pull.service
  • mnt-cdn.mount have to be active (mounted)
  • cdn-pull.service should be active (script is running) or inactive (sync is completed). In both cases, it’s ok.

With this set-up, when you restart the mnt-cdn.mount files will be automatically pulled and synced to RAM when system starts and when you start or restart mnt-cdn.mount service.

Next you can serve these files on nginx and the final step could be to auto-detect push to update files automagically.

See also