Clone virtual machines to keep a pristine image

The basic idea of using virtual machines for testing is to always start from a clean state, as close as possible to what a user would get after freshly installing the system on their real machines.

So, every time we need to test something, we could create a new virtual machine and intall Ubuntu from scratch there, but that would take a considerable amount of time.

An alternative is to install a machine once, and keep it clean. Never test in that one, and instead clone it every time you need to test something. Use the clone to experiment, and delete it when you are done. Cloning the machine is a lot faster than reinstalling each time. The base machine, the one you clone everytime, is called a pristine machine. If you are careful with it, the only time you will need to reinstall is when you are testing the installer.

Cloning the pristine machine using Virtual Machine Manager is really simple, it’s done in like two clicks.

First, of course, you will need to install your pristine machine. I always put pristine in the name, to make it less likely that I will start using it for testing. Every time I forget and play with the pristine machine, I’m polluting its state, and have to recreate it.

With the pristine machine ready, I then recommend to open it and run in a terminal:

  sudo apt update && sudo apt upgrade

That will update all the packages installed, so your clone starts also in the most up-to-date state. If you do this step every time, you will never have to wait for long while your machine downloads and installs lots of packages, just wait a little every day.

image

Now shut down the pristine machine and don’t touch it again, only to update it.

To clone it, right click on it and then click the Clone... button.

image

A dialog will be opened, where you can enter the new virtual machine details.

image

I always write the date as part of the name, to remember when I created it. But the most important thing to do here is to make sure that the Storage option is set to Clone this disk. Otherwise, you will be sharing the disk with the pristine machine, which will pollute its state, the exact thing we are trying to avoid.

This will take some time while the whole disk is copied. But not long, and once it’s done you can freely play and pollute this clone, without affecting the pristine machine.

Ubuntu Testing Day wrap up - 20161125

Today we had our first testing day. We will keep doing this every Friday, and at the end of the session I will post a summary with links to follow up and learn more about the subject in case somebody is interested.

You can watch the video by clicking the image below.

Alt text

For this session we had Kyle Fazzari talking about his work as the maintainer of the Nextcloud snap.

You can find more info about the project in the Nextcloud main page and more info about the snap in Kyle's blog.

We tested the snap using a virtual machine. To set one up you can use the following guides:

After the virtual machine is ready, Nextcloud can be installed in there by just running in the terminal:

$ sudo snap install nextcloud

If you want to test an unreleased and unstable version to help the Nextcloud developers, you can add the flag --edge or --beta to that command.

I wrote some steps to guide you getting started with the testing in this gist.

And finally, if you want to join our community, we are usually in Rocket Chat. You can join the #community, #qa or #snapcraft channels, or any other that catches your attention.

Special thanks to Julia and CoderEurope for joining us during the session. And of course to Kyle and the Nextcloud community for this amazing piece of free software that encourages everybody to take control over their data, in a really simple way.

You are all invited to join us the next time, which will be Friday, December 2nd, again at Ubuntu On-Air. The time and theme will be announced soon. Or at least before it starts.

Install Ubuntu in a virtual machine

There are multiple ways to test Ubuntu. The most simple, and most dangerous is to just use your real Ubuntu machine to install and try things. I wouldn’t recommend this one to begin with, because we will be using experimental software that it’s likely to completely break your operating system every now and then.

This is not so terrible. Unless you are dealing with low-level stuff, the worst case scenario will be solved by reinstalling Ubuntu. And usually it won’t even come to that. My machines always have the unreleased unstable version of Ubuntu, and I’m often learning interesting things by trying to recover the mess I make. But let’s not get to that yet, let’s first see other options.

My preferred way to test graphical applications is to use Kernel-based Virtual Machine (KVM) and Virtual Machine Manager hosted on my real machine. A virtual machine emulates a real machine. It is completely isolated from the host where it runs, so you can freely experiment in it without the risk of breaking a machine that you will later need for something else. If things go crazy, just throw away the virtual machine and start again. The downside is that it’s slower than testing in some of the alternatives, like LXC or Docker. But I like it because it gives me an environment that’s really close to what a user would get when he installs Ubuntu on his machine. Also it’s really simple with the Virtual Machine Manager.

For this I’m going to assume that you have a machine with Ubuntu already. If you don’t have one, you can follow this guide to install Ubuntu by yourself, or you can contact somebody from your Local Community Team using the LoCo Portal. There’s always somebody who would like to meet and help a new Ubuntero or Ubuntera to get started. And in case of doubt, just ask your question in Ask Ubuntu and you will get quick help there too.

To install everything, let’s open a terminal, like Mr. Robot. Click the Ubuntu icon on the top-left of the screen, type terminal and click on the first result.

image

Many of the things we do with commands in the terminal can also be done using graphical applications. The terminal might seem overcomplicated at first, but soon you will notice that it’s easier to get lost trying to click buttons. In here we will do both; for some things I like commands, for some I like buttons. However, the first rule is to never enter a command that you don’t understand; and you should be specially suspicious if the command asks for your password. If you don’t understand what you are doing, search and ask.

Now, on the terminal, type:

  sudo apt install qemu-kvm libvirt-bin bridge-utils virt-manager

And press enter.

sudo is the command to execute other commands as a super user. You will need super user permissions to do administrative actions on the machine, like installing new software. apt is used to manage packages. So in here we are telling the machine to install 4 packages using apt as an administrator. The 4 packages are the software we will use to create and manage virtual machines.

The first thing that sudo will do is to ask for your password.

image

Again, make sure that you are only following command line instructions from a trusted source, like me ^_^ And make sure that you know what you are doing before pressing enter.

Type your user password in the terminal and press enter. The password will be hidden so nobody else can see it while you are typing it, but don’t worry, after pressing enter the command will proceed with the installation.

You will know that it finished because you will be presented with a prompt again, something like user@machine:~$ which indicates that the terminal is ready to receive a new command.

Now let’s jump back to the graphical user interface. Click again the top-left Ubuntu icon, type virtual machine manager and click the first result.

The Virtual Machine Manager window will be opened. So let’s create a new machine clicking the top-left shiny icon.

image

This will open a dialog to choose the installation method. We will install Ubuntu from an ISO image file. If you don’t have one, you can follow the previous guide to download the latest Ubuntu version with Long Term Support. Once you get it, select the Local install media option and click the Forward button.

image

In the next dialog page, click the Browse button, which will open a new dialog. In the new one, click Browse Local, which will open yet another dialog where you should select the downloaded Ubuntu ISO file (and now you see why I said that the command line was often easier :) Back at the first dialog, clilck the Forward button.

image

Next step is to select the amount of resources that you want to give to this virtual machine. If you give it a lot of resources, it will be faster, but then your real machine will be slower. So you have to balance both, another downside of using full virtual machines instead of ligher ways to simulate a machine. For your first machine, maybe just leave the default values. Pay attention at how the virtual and real machine feel, and next time choose a different value to compare.

image

So, Forward one more time and now we can choose the size of the disk of the virtual machine. I have found that 20GiB is generally enough, and you will throw away your machine long before you fill the space. Less than that and you will likely get the disk full and will have to resize it, which is kind of boring. I will paste the command for that some other day. Forward!

image

And this is the last step, at least for this post. Enter a name for the virtual machine and click Finish.

Here the interesting part begins, finally. You will see the virtual machine booting, just like a real one does. It will start the live Ubuntu system, which runs from volatile memory, so you have to install it in the hard disk of the virtual machine to make it permanent. And then again, as it emulates a real machine, the steps to install Ubuntu in a virtual machine are the same to install it in a real one, so the same https://www.ubuntu.com/download/desktop/install-ubuntu-desktop guide to install a desktop machine that I linked earlier will be useful.

Once the virtual machine reboots, take some time to play with the user interface of Virtual Machine Manager. You will find controls to start, shutdown and reset the machine, and some more advanced options to change the hardware it is emulating. Remember that you can’t break your real machine, so play, make a mess, and start again.

If you want to learn some more about Ubuntu, and collaborate with the community, join us in the Ubuntu Testing Days, every Friday in Ubuntu On Air. We will use virtual machines and other tools to test cool free software projects.

Download the latest Ubuntu LTS

The Ubuntu Long Term Support (LTS) version is released every two years, on April; and it is supported for five years. So this version has to be super stable, it’s what most of our users have installed in their machines, and it often gets security fixes and some other stable release updates.

All of this means that it always needs to be tested, and by as many hands and on as many different machines as possible. If you want to start helping the Ubuntu community, I would suggest that testing the latest LTS is the place to start. There will be so many people around the world that will be positively affected by this work, and all of them will sincerely appreciate it. Well... not all of them, but most of them for sure :) I for one will invite you to a beer when you come for holidays to Costa Rica.

While testing Ubuntu you will learn about the many different aspects that build a free operating system. Then you can jump to collaborate in other areas that pique your interest, like support, programming, translations, packaging, planning, partying, etc. Or you can stay helping with the tests, maybe in some more unstable and rapidly changing projects.

This is the first post of a series I will write about contributing to different areas of Ubuntu. They will focus mostly on testing because that’s what I enjoy the most. And in addition, we will meet live every Friday on Ubuntu On-Air to explore cool free software projects and help the developers on their way to a stable release.

Let’s begin. The first thing you will have to do if you want to join our community is to download Ubuntu. To do that open your browser and go to https://www.ubuntu.com/download

Click the Ubuntu Desktop link, or the download button next to it.

image

There are other different versions of Ubuntu. Newer and older releases, different user interface flavors, smaller versions for servers and connected devices, and some for specific phones and tablets. We will see some of them in following posts and meetings, but for now we will start with the basic and most common.

The next step shows you the recommended requirements that a machine should meet to run this Ubuntu version properly. Make sure that you have all of them and click the Download button.

image

This opens a page for optional donations. If you feel generous, you can give some money to the community and decide where it should be spent. If you can’t or don’t want to give any money, no problem, just click the link that says: Not now, take me to the download. Beers, hugs, bugs are all valid payment methods too.

image

And that’s it. A dialog should appear offering you to download the file. Make sure to choose the option to save it into your disk, and take into account that if you use a different browser than mine, your dialog might look different.

image

This was a first simple step. If it was too simple for you, don’t worry, next time we will do a few more complicated things. If on the other hand you got lost somewhere, also don’t worry because we will be around to answer any kind of doubts.

In both cases, come and join us next Friday, November 25th at 18:00 UTC on our first Ubuntu Testing Day! We will use this image to test the awesome Nextcloud snap. Kyle Fazarri has been making some great work to package Nextcloud for everybody, and he will be joining us to show off and answer questions.

On writing my first snaps, and keeping my passwords secure everywhere

I’ve been writing snaps for a year, more or less. But all the snaps I had written were a one-time thing to reproduce a bug, automate a scenario, or explore a specific code path. A few months ago I decided to write one snap that I would use day-to-day. The application I use the most by far is keepass, so that seemed like a good place to start.

kpcli

With kpcli I access my password database from the command line. It’s just one big perl script, but not as ugly as it sounds :) That was a great first experiment because I haven’t tried perl in a snap before, and it would let me access the passwords from my beagle bone black in a secure and confined way.

Because it is a script, I just need to copy it into the snap. Of course, there’s a plugin for that! snapcraft is a meta-packaging tool to generate snaps. It comes with plugins for many packaging tools, copy being the most simple of them. Run:

  $ sudo apt install snapcraft
  $ snapcraft help copy

The next step is to look at the dependencies. Luckily, they are right there in the README file, in the form of Ubuntu packages. snapcraft calls them stage-packages, and they will be installed into the snap during the build.

Now I have almost all the difficult details to make the snap, and it was surprisingly easy. With this information, I wrote a part (read more about parts):

  kpcli:
    source: https://github.com/alecsammon/kpcli.git
    plugin: copy
    files:
      kpcli.pl: scripts/kpcli.pl
    stage-packages:
      - perl-base
      - libcrypt-rijndael-perl
      - libterm-readkey-perl
      - libsort-naturally-perl
      - libfile-keepass-perl
      - libterm-shellui-perl
      - libterm-readline-gnu-perl
      - libclone-perl

I am just missing a way to tell the command where to find the perl libraries that I am bundling inside the snap. There is work in progress to make it easier to set up environment variables before launching the command; but for my first simple snap, quick and easy will do. I wrote a wrapper script:

  #!/bin/sh

  $SNAP/usr/bin/perl -I $SNAP/usr/share/perl/5.22 -I $SNAP/usr/lib/x86_64-linux-gnu/perl/5.22 -I $SNAP/usr/lib/x86_64-linux-gnu/perl5/5.22 -I $SNAP/usr/share/perl5 $SNAP/scripts/kpcli.pl

and the corresponding part:

  launcher:
    plugin: copy
    files:
      run.sh: scripts/run.sh

Finally, I added details about the name, version, summary and description of the snap, and defined the kpcli app. You can download the source of this snapcraft.yaml, put it in a directory and run snapcraft in there to generate it yourself. Or try mine installing it from the store:

  $ sudo snap install kpcli-elopio
  $ kpcli-elopio.kpcli

KeepassX

Ok, that solves command line access, but I also want some eye-candy, sometimes. KeepassX gives me access to the same passwords database through a GUI. By the time I finished my CLI snap I saw work from kyrofa, sergiusens and dpm to make it easy to snap desktop apps and use them in Ubuntu classic. I also was at a sprint sitting next to attente, who knows everything I didn’t know about the desktop, so it was a great time to enter the world of collaborative snapping.

Let me digress here for a moment to talk about collaboration. snapcraft and snapd are free software, and a lot of their design is based on the idea that the system can be extended and improved through the shared work of many and diverse developers. snapcraft aims to be the universal packaging tool, and plugins are the way we integrate the wild variety of build systems into the lifecycle to generate a snap. KeepassX is more complex than the CLI client, and it uses cmake to generate the executable. Well, guess what? There’s a plugin for that! :) (see the source of the cmake plugin). Also, this application uses the QT5 UI libraries. Just like with the kpcli snap, that means I will require to set up some environment variables and a launcher; but somebody else already did that and we have a way to share and reuse parts. Run:

  $ snapcraft update
  $ snapcraft search qt5
  $ snapcraft define qt5conf

There’s a part for that!! And when this application is running, I would like it to be integrated with unity7, if available. That means I would need a few extra permissions for my isolated app to access the unity7 services and libraries installed in a classic Ubuntu system. I’m a lucky guy because there’s an interface for that!!! (see the source of the unity7 interface) By default, snaps will be fully confined and won’t be able to access anything they don’t provide by themselves. Through interfaces, we allow snaps to get extra permissions to consume services, access devices and directories, and do anything that you could do in a classic unconfined Linux system; but now in a controlled manner with the explicit acknowledgement of the user for sensitive stuff. Run:

  $ snap interfaces

We have plugins, parts and interfaces for the most common cases of the most common apps. But the fun is just starting, and there are lots and lots of apps out there that would benefit from the controlled dependencies, usable security and transactional updates of snaps. We celebrate when we get new people into the team, so if you want to participate and influence the future of this shiny project, we can help you getting started in the playpen.

One last detail about collaboration is the relation with upstreams and distribution maintainers. Snapping free software projects is a pleasure because the source is available for us to inspect and decide on the best strategy to build the snap. Then we spread the apps to more people, we get more feedback and in many cases we end up sending patches to improve the upstream project. And I already mentioned the use of Ubuntu packages during the build of the snaps. With this we bootstrap from the awesome work that the Debian and Ubuntu communities are doing, and we’ll keep benefitting from both sides as we need debs to build some snaps, and the two play nicely together.

Aaaand back on track, because with all the work shared by the community this KeepassX snap is almost done by itself, but not quite.

I start by creating a new snapcraft.yaml file an defining the snap details:

  name: keepassx-elopio
  version: "2.0.2"
  summary: KeePassX is a cross platform password safe
  description: |
    KeePassX is an application for people with extremly high demands on secure
    personal data management. It has a light interface, is cross platform and
    published under the terms of the GNU General Public License.

Next, the app:

  apps:
    keepassx:
      command: qt5-launch keepassx
      plugs: [unity7, opengl]

Take a look at that qt5-launch, which is provided from the shared qt part that I will reference in a few lines. keepassx is a binary that lives inside the snap, and that we will compile using a snapcraft plugin. Also look at those plugs. That’s how we tell the system that a snap wants to consume an interface slot of the same name.

Followed by the parts:

  parts:
    keepassx:
      source: https://github.com/keepassx/keepassx.git
      source-tag: 2.0.2
      plugin: cmake
      build-packages:
        - g++
        - qtbase5-dev
        - libqt4-dev
        - libqt5x11extras5-dev
        - qttools5-dev
        - qttools5-dev-tools
        - libgcrypt20-dev
        - zlib1g-dev
      stage-packages:
        - libgtk2.0-0
        - libqt5gui5
        - unity-gtk2-module
        - appmenu-qt5
        - dconf-gsettings-backend
      after: [qt5conf]

Note the new build-packages section. Those packages will not end up in the snap, they are only used during the compilation so they will be installed in the host where you are running snapcraft. I already discussed about the stage-packages, nothing new to see there. Now the after keyword is where a lot of the collaboration magic happens. It means that my keepassx part depends on the qt5conf part. But I don’t define the qt5conf part in my yaml file, so snapcraft will bring it from the pool of shared parts (see the source of the qt5conf part). I love this because I didn’t have to write any of the qt complications, and when a qt expert makes a change to improve it, I just need to run snapcraft again and my snap will be better with no effort. This is where the qt5-launch I used in the apps section comes from.

I finish my yaml adding a last part:

  ...
  parts:
    keepassx:
      ...
    glibcompiledassets:
      plugin: copy
      files:
        gschemas.compiled: usr/share/glib-2.0/schemas/gschemas.compiled
        giomodule.cache: usr/lib/x86_64-linux-gnu/gio/modules/giomodule.cache

This is required to save the KeepassX settings. It’s currently not a nice way to include the compiled files in the snap, and we are discussing about how to improve it, so I won’t dig too much here. It shows how simple yet flexible snaps are: you can put anything into the snap, it doesn’t matter how you got it. But it also shows that we have things to improve, so let me insist on my invitation for you to join our project. If there’s something you hate, this is the right time to change it. If there’s something you like, there’s always room to improve it. We have cookies.

Punto final, café con tamal. Give it a try:

  $ sudo snap install keepassx-elopio
  $ keepassx-elopio.keepassx

image

Convert cbr files to pdf

Here’s a quick tip. Generally when some of my good internet pirate friends share with me a comic book,1 it comes in the cbr format. There is no application for my Ubuntu phone to read these files (yet), but there are some good pdf readers. All I have to do is convert the files with some handy terminal commands.

A cbr file is just a group of image files compressed in the rar format. So, let’s uncompress the files in a temporary directory:

  dir=$(mktemp -d)
  cd $dir
  unrar x /path/to/file.cbr

And then let’s convert them to pdf:

  convert * /path/to/file.pdf

Open the pdf to check it is OK:

  evince /path/to/file.pdf

And delete the temporary directory:

  cd ..
  rm -rf $dir

Transfer to the phone, and enjoy.

Sharing the Wi-Fi connection through Ethernet

My apartment was clearly not designed for the Internet of Things. I have a weird set up to connect all my devices: there’s ADSL, Ethernet, Wi-Fi, powerline networking, 4G from a free promotion until I download 3GB and then 3G for the rest of the month... It’s crazy, and getting crazier as I have to test more and more devices for snappy. When I was about to throw a cable from the kitchen to the office I found about sharing the network (thanks Luis!). Now let me repeat a slightly modified version of part of Luis’ answer to show how I share my wireless connection through an ethernet cable with my test board.

  1. Open the network indicator.

  2. Click Edit Connections....

    image

  3. On the Network Connections dialog, click the Add button.

    image

  4. On the Choose a Connection Type dialog, select Ethernet.

    image

  5. On the Editing dialog, enter a name for the connection.

  6. Go to the IPv4 Settings tab.

  7. Select the Method Shared to other computers.

    image

  8. Click the Save button.

What’s left is to connect an Ethernet cable from your laptop to the board, give power to the board and wait for it to finish booting.

To get the IP of the board you can run the arp command (Thanks to Alex for the tip). It will show you the addresses of the neighbour machines. The one of your board will be like 10.42.0.?. Now you can ssh into the board using the default credentials (username ubuntu, password ubuntu) or the ones you configured.

image

This also proved to be useful during our first snappy maker night, when we had to connect many boards to play with the system.

Pro tip: If you are having problems connecting through ssh, you can connect through the serial console to check for errors.

Connecting to Snappy through the serial console

I’m working on Snappy Ubuntu with a BeagleBone Black Rev C and a Raspberry Pi 2. You can connect to the boards through the serial console to watch for errors during the boot process and to control the system. After my first burned cable I decided to write down the instructions to connect the cables to reduce the likelihood of my boards catching fire.

First, of course, you should flash the SD card with snappy. Then, get one of these FTDI cables that convert USB to serial. As this cable has the plugs separated, it will work for both boads. The black wire is ground (GND), the white wire is receive into the USB port, the green wire is transmit out of the USB port and the red wire is 5V power.

BeagleBone Black

The BeagleBone Black Rev C has 6 pins. There’s a white dot next to the first pin. Connect the black GND wire to the first pin, the green wire to the fourth pin and the white wire to the fifth pin. On the BeagleBone NEVER connect the red power wire.

image

image

Raspberry Pi

The Raspberry Pi 2 Model B has 40 pins. Here you can find a nice pinout drawing with the numbers and functions of the pins.

Connect the black GND wire to the sixth pin, the white wire to the eight pin and the green wire to the tenth pin. You can optionally power the Raspberry connecting the red wire to the second pin, but if you do this DO NOT connect the USB power connector as you can’t have both power sources.

image

image

Serial console

Install screen:

  sudo apt-get install screen

Plug the other end of the cable to an USB port of your computer and figure out the name of the tty device:

  ls /dev/ttyUSB*

And start the screen session on the serial terminal, replacing the /dev/ttyUSB? with the number of your device.

  sudo screen {/dev/ttyUSB?} 115200

Finally, insert the SD card and plug in the power source of the board. You will see the boot messages on your terminal and in the end you will be presented with the login prompt. The default user is ubuntu and the default password is ubuntu too.

image

Snappy has an SSH server preinstalled, so once you booted successfully you can login through SSH, which will give you a better experience. You can use the serial console to query the IP address of the board:

  ip addr show

And from a different terminal, replacing ip with the board IP address:

  ssh ubuntu@{ip}

Now you are ready to start playing with snappy on the board. Don’t miss the tour.

To exit the serial console, press CTRL+a and then k.

More information about the cables and the serial console

Awful Shape, what art thou?

JUPITER
[...]
Awful Shape, what art thou? Speak!

DEMOGORGON
Eternity – demand no direr name.
Descend, and follow me down the abyss;
I am thy child, as thou wert Saturn’s child,
Mightier than thee; and we must dwell together
Hencefoth in darkness. –Lift thy lightnings not.
The tyranny of Heaven none may retain,
Or reassume, or hold succeeding thee...
Yet if thou wilt–as ’tis the destiny
Of trodden worms to writhe til they are dead–
Put forth thy might.

on Prometheus Unbound, Act III, Scene I

There is no God

I was an infant when my mother went
To see an atheist burned. She took me there:
The dark-robed priests were met around the pile;
The multitude was gazing silently;
And as the culprit passed with dauntless mien,
Tempered disdain in his unaltering eye,
Mixed with a quiet smile, shone calmly forth:
The thirsty fire crept round his manly limbs;
His resolute eyes were scorched to blindness, soon;
His death-pang rent my heart! the insensate mob
Uttered a cry of triumph, and I wept.
“Weep not, child!” cried my mother, “for that man
Has said, ’There is no God.’”

on Queen Mab, Canto VII