_____100 |> F# Some Troubleshooting Linux

In the last article I wrote on writing a code kata with F# on OS-X or Windows, I had wanted to use Linux but things just weren’t cooperating with me. Well, since that article I have resolved some of the issues I ran into, and this is the log of those issues.

Issue 1: “How can I resolve the “Could not fix timestamps in …” “…Error: The requested feature is not implemented.””

The first issue I ran into with running the ProjectScaffold build on Linux I wrote up and posted to Stack Overflow titled “How can I resolve the “Could not fix timestamps in …” “…Error: The requested feature is not implemented.”“. You can read more about the errors I receiving on the StackOverflow Article, but below is the immediate fix. This fix should probably be added to any F# Installation instructions for Linux as part of the default.

First ensure that you have the latest version of mono. If you use the instructions to do a make and make install off of the fsharp.org site you may not actually have the latest version of mono. Instead, here’s a good way to get the latest version of mono using apt-get. More information can be found about this on the mono page here.

apt-get install mono-devel
apt-get install mono-complete

Issue 2: “ProjectScaffold Error on Linux Generating Documentation”

The second issue I ran into I also posted to Stack Overflow titled “ProjectScaffold Error on Linux Generating Documentation“. This one took a lot more effort. It also spilled over from Stack Overflow to become an actual Github Issue (323) on the project. So check out those issues in case you run into any issues there.

In the next issue, to be published tomorrow, I’ll have some script tricks to use mono more efficiently to run *.exe commands and get things done with paket and fake in F# running on any operating system.

Ubuntu

Starting an Ubuntu Dev Tools List

I’ve recently setup a completely clean virtual machine for doing web, system, and related development on Ubuntu. Here’s the shortlist of what I’ve installed after a default installation. The ongoing list of tools and related items I have installed on my Linux dev box I’m keeping here, and it will be kept as a living doc, so I’ll change it as I add new tools, apps and related changes. So lemme know what I ought to add to that list and I’ll add it to my docs page here. Here’s what I have so far…

Other To-dos

  • Always run sudo apt-get update once the system is installed. It never hurts to have the latest updates.
  • I always install Chrome as my first app. Sometimes the Ubuntu Software Center flakes out on this, but just try again and it’ll work. I use the 64-bit Chrome btw, as I’ve noticed that the 32-bit often flakes out when attempting installation on my virtual machines. Your mileage may vary.

What this enables…

At this point I can launch into about any language; Java, JavaScript, and a few others with a minimal amount of headache. Since it’s a Linux instance it gives me a full range of Linuxy things at my disposal.


Default Java Installation

  1. Run a ‘sudo apt-get update’.
  2. To install the default Java JRE and the JDK run the following commands.
    sudo apt-get install default-jre
    sudo apt-get install default-jdk


Oracle Java v8 Installation

  1. sudo add-apt-repository ppa:webupd8team/java
    sudo apt-get update
    sudo apt-get install oracle-java8-installer


WebStorm Installation

  1. I download the application zip from JetBrains and then run
    tar xfz WebStorm-*.tar.gz
  2. Next I always move the unzipped content to the directory in which I’d like to have the application stored. It’s good practice to not keep things in the download directory, just sayin’. Generally I put these in my usr/bin directory.
    mv /downloads/WebStorm-* your/desired/spot
  3. Now at your terminal, navigate to the path where the application is stored and run the WebStorm.sh executable.
    ./bin/webstorm.sh
  4. To add WebStorm to the Quicklaunch, just right click on the icon and select to Lock to Launcher.


IDEA IntelliJ Installation

  1. Follow all the steps listed under WebStorm, it’s the exact same process.


Sublime 3

  1. Go to download the latest v3.
  2. Run the package and it should launch the actual Ubuntu installer, setup Sublime for bash use and get it installed.

(NOTE UPDATED 1/18/2016 > The installer doesn’t seem to get it installed, so I went with this link http://olivierlacan.com/posts/launch-sublime-text-3-from-the-command-line/ which has a good solution.)

Learning About Docker

Over the next dozen or so few days I’ll be ramping up on Docker, where my gaps are and where the project itself is going. I’ve been using it on and off and will have more technical content, but today I wanted to write a short piece about what, where, who and how Docker came to be.

As an open source engine Docker automates deployment of lightweight, portable, resilient and self-sufficient containers that run primarily on Linux. Docker containers are used to contain a payload, encapsulate that and consistently run it on a server.

This server can be virtual, on AWS or OpenStack, in clusters, public instances or private, bare-metal servers or wherever one can get an operating system to run. I’d bet it would show up on an Arduino cluster one of these days.  😉

User cases for Docker include taking packaging and deployment of applications and automating it into a simple container bundle. Another is to build PaaS style environments, lightweight that scale up and down extremely fast. Automate testing and continuous integration and deployment, because we all want that. Another big use case is simply building resilient, scalable applications that then can be deployed to Docker containers and scaled up and down rapidly.

A Little History

The creators of Docker formed a company called dotCloud that provided PaaS Services. On October 29th, 2013 however they changed the name from dotCloud to Docker Inc to emphasize the focus change from the dotCloud PaaS Technology to the core of dotCloud, Docker itself. As Docker became the core of a vibrant ecosystem the founders of dotCloud chose to focus on this exciting new technology to help guide and deliver on an ever more robust core.

Docker Ecosystem from the Docker Blog. Hope they don't mind I linked it, it shows the solid lifecycle of the ecosystem. (Click to go view the blog entry that was posted with the image)

Docker Ecosystem from the Docker Blog. Hope they don’t mind I linked it, it shows the solid lifecycle of the ecosystem. (Click to go view the blog entry that was posted with the image)

The community of docker has been super active with a dramatic number of contributors, well over 220 now, most who don’t work for Docker and they’ve made a significant percentage of the commits to the code base. As far as the repo goes, it has been downloaded over a 100,000 times, yup, over a hundred. thousand. times!!! It’s container tech, I’m still impressed just by this fact! On Github the repo has thousands of starred observers and over 15,000 people are using Docker. One other interesting fact is the slice of languages, with a very prominent usage of Go.

Docker Language Breakout on Github

Docker Language Breakout on Github

Overall the Docker project has exploded in popularity, which I haven’t seen since Node.js set the coder world on fire! It’s continuing to gain steam in how and in which ways people deploy and manage their applications – arguably more effectively in many ways.

Portland Docker Meetup. Click image for link to the meetup page.

Portland Docker Meetup. Click image for link to the meetup page.

The community is growing accordingly too, not just a simple push by Docker/dotCloud itself, but actively by grass roots efforts. One is even sprung up in Portland in the Portland Docker Meetup.

So Docker, Getting Operational

The Loading Bay

The Loading Bay

One of the best ways to describe docker (which the Docker team often uses, hat tip to the analogy!) and containers in general is to use a physical parallel. One of the best stories that is a great example is that of the shipping and freight industry. Before containers ships, trains,

Manually Guiding Freight, To Hand Unload Later.

Manually Guiding Freight, To Hand Unload Later.

trucks and buggies (ya know, that horses pulled) all were loaded by hand. There wasn’t any standardization around movement of goods except for a few, often frustrating tools like wooden barrels for liquids, bags for grains and other assorted things. They didn’t mix well and often were stored in a way that caused regular damage to good. This era is a good parallel to hosting applications on full hypervisor virtual machines or physical machines with one operating system. The operating system kind of being the holding bay or ship, with all the freight crammed inside haphazardly.

Shipping Yards, All of a Sudden Organized!

Shipping Yards, All of a Sudden Organized!

When containers were introduced like the shiny blue one shown here, everything began a revolutionary change. The manpower dramatically

A Flawlessly Rendered Container

A Flawlessly Rendered Container

dropped, injuries dropped, shipping became more modular and easy to fit the containers together. To put it simply, shipping was revolutionized through this invention. In the meantime we’ve all benefitted in some way from this change. This can be paralleled to the change in container technology shifting the way we deploy and host applications.

Next post, coming up in just a few hours “Docker, Containers Simplified!”

Linux Containers

Linux Containers, LXC, FreeBSD Jails, VServer…

These days containerization of work, applications and storage on systems has become a hot topic. Not to say it wasn’t before, but it’s got a boost from the cloud computing segment of the industry. With that I felt the need to write up what I’ve discovered of the history in this industry so far. I’d love feedback and corrections if I’ve got anything out of order here or if – heaven forbid – I’ve got something wrong.

What are Containers?

Before I get into what a container is, it is best to define what operating system-level virtualization is. Sometimes this is referred to as jailed services or apps running in a jail.

This level of virtualization often provides extremely similar functionality as a VMware, Virtual Box or Hyper-V virtual server would provide. The difference however is primarily around the idea that the operating system-level virtualization actually runs as a service, usually protected, that runs apps as if it were an operating system itself.

So what’s a container?

Linux Contains is a feature that allows Linux to run a single or more isolated virtual systems that each have their own network interfaces, computer process threads and namespaces, user namespaces and states.

One of the common abbreviations for Linux Containers you’ll see is LxC. There are however many distinct operating system-level virtualization solutions.

  • Open VZ – this technology uses a single patched Linux kernel, providing the ability to use the architecture and kernel version of the system that is executing the container.
  • Linux V-Server – this technology is a virtual private server implementation that was created by adding operating system-level virtualization to the Linux kerne. The project was started by Jacques Gélinas. It is now maintained by Herbert Pötzl of Austria and is not related to the Linux Virtual Server project. The server breaks things into partitions called security contexts, within that is the virtual private server.
  • FreeBSD Jail – This container technology breaks apps and services into jails.
  • Workload Partitions – This is a technology built for AIX, introduced in AIX 6.1. Workload Partitions breaks things into WPARs. These are software partitions that are created from the resources of a single AIX OS instance. WPARs can be created on any system p (the new old thing, was the RS/6000 tech) hardware that supports AIX 6.1 or higher versions. There are two kinds of WPARs, System WPARs and Application WPARs.
  • Solaris Containers – is a container tech for x86 and SPARC systems. It was first released in February 04′ for Solaris 10. It is also available in OpenSolaris, SmartOS and others as well os Oracle Solaris 11. The Solaris container combines resource controls in seperations referred to as zones. These zones act as completely isolated virtual servers within a OS.

What is so great about a container?

Ok, so I’ve covered what a container is. You’re probably asking, “so what do I do with these containers?” There are a number of things, for starters speed is a huge advantage with containers. You can spool up entire functional application or service systems, like an API facade or something, in seconds. Often times a container will spool up and be ready in less than a second. This provides a huge amount of power to build out flexible, resilient, self-healing distributed systems that otherwise are just impossible to build with slow loading traditional virtual machine technology.

Soft memory is another capability that most containers have. This is the capability of being allocated, or being allocated and running, in memory. As one may already know, if you run something purely out of memory it is extremely fast, often 2-10x faster than running something that has to swap on a physical drive.

Managing crashing services or damaged ecosystem elements. If the containers are running, but one gets hit with an overloaded compute ask, software crashes on it, or one of the many receive some type of blocking state like a DDOS of sorts, just reboot it. Another option is just to kill it and spool up and entirely new instance of the app or service in a container. This ability really is amplified in any cloud environment like AWS where a server instance may crash with some containers on it, but having another instance running with multiple containers on it is easy, and restarting those containers on running instances is easy and extremely fast.

Security is another element that can be assisted with container technology. As I alluded to in the previous point above, if a container gets taken over or otherwise compromised, it’s very easy to just kill it and resume one that is not compromised. Often buying more time to resolve the security concern. Also, by having each container secured against each other container, controlling a container does not result in a compromised physical machine and operating system. This is a huge saving grace when security is breached.

Container Summary

Containers are a hot ticket topic, for good reason. They provide increase management of apps and services, can utilize soft memory, increase security and they’re blazing fast. The technology, albeit having been around for a good decade, is starting to grow in new ways. Containers are starting to also become a mainstay of cloud technology, almost a requirement for effective management of distributed environments.

Next up, I’ll hit on Docker tech from DotCloud and Salomon Hykes @solomonstre.

For now, anybody got some additions or corrections for this short history and definitions of containers?  🙂

Containers

Getting Docker Installed on Ubuntu 12.04 LTS

A few days ago I posted the blog entry “Using SSH Locally to Work With Ubuntu VM + VMware Tools Installation via Shell“, it was related to getting a clean Ubuntu Server install running with VMware Tools and so that I could use it as a hosted instance. Simply put, being able to SSH into it just as I would a hosted AWS or Windows Azure Ubuntu Server image. Once I had the default virtual machine running 12.04 LTS I went about another installation that is needed to run Docker. Docker will have issues with anything pre-3.8 kernel. Running the command below will show that kernel 3.5 is the current kernel in 12.04 LTS.

apt-cache search linux-headers-$(uname -r)

To update to the 3.8 kernel I ran the following command and then rebooted.

sudo apt-get install linux-image-generic-lts-raring linux-headers-generic-lts-raring
sudo shutdown -r now

With the reboot complete, I checked the kernel version again and 3.8 was installed successfully.

@ubuntu:~$ apt-cache search linux-headers-$(uname -r)
linux-headers-3.8.0-33-generic - Linux kernel headers for version 3.8.0 on 64 bit x86 SMP

To get Docker installed (as of 0.6) run the following command.

sudo sh -c "wget -qO- https://get.docker.io/gpg | apt-key add -"
sudo sh -c "echo deb http://get.docker.io/ubuntu docker main\
> /etc/apt/sources.list.d/docker.list"

Next update the sources, then install lxc-docker.

sudo apt-get update
sudo apt-get install lxc-docker

To verify that docker is installed I executed the following command and…

sudo docker run -i -t ubuntu /bin/bash

…see similar results just after issuing the command.

Unable to find image 'ubuntu' (tag: latest) locally
Pulling repository ubuntu
8dbd9e392a96: Download complete
b750fe79269d: Download complete
27cf78414709: Download complete

After that displays then I typed exit to leave docker. I now have a running version of docker on the Ubuntu 12.04 LTS instance ready for testing and hacking with docker.

Using SSH Locally to Work With Ubuntu VM + VMware Tools Installation via Shell

I do a lot of work with Ubuntu, 90% or so of that work is from an Ubuntu instance. Often that instance happens to be a local VM running in VMware Fusion (or sometimes Virtual Box). Often I’ll start with a base server image which isn’t entirely setup for SSHing into the instance. These are the steps to get that installed and ready to go.

First install the image, in this particular situation I’m using the Ubuntu 12.04 LTS Server image.

Ubuntu 12.04 Server. Click for full size image.

Ubuntu 12.04 Server. Click for full size image.

That will take a few minutes to install, on machines these days I’ve experience just about 8-15 minutes. There are a million other options to do this too, such as starting with a clean Ubuntu image using Vagrant, which takes all of about 1-2 minutes, sometimes a bit more if you have to download the image. But either way, get one built and running.

Installing Ubuntu using VMware Fusion. Click for full size image.

Installing Ubuntu using VMware Fusion. Click for full size image.

Once the image is installed, login and install openssh-server and openssh-client.

sudo apt-get install openssh-server openssh-client

Once that’s installed I pull up my IP address with ifconfig.

ifconfig

The ifconfig command shows a lot of information regarding the network configuration associated with the various network adapters in the machine that it is executed on. In the image I’ve circled the local IP address that is assigned to the instance.

The local IP address using the ifconfig command. Click for full size image.

The local IP address using the ifconfig command. Click for full size image.

Now that you have the local IP of the instance, bring up a local terminal (in this case I’m on OS-X, but if you’re on Windows pull up Putty or on Linux or another *nix variant pull up a shell). In the terminal you can now enter the follow SSH command to log in from the local machine versus the running instance. This comes in handy when you want to treat the machine like an actual hosted machine somewhere, in which you wouldn’t be directly logged into the server.

ssh username@192.168.77.197
Logged In.

Logged In.

Getting VMware Tool Installed

This assumes that you mount the installation files (aka the cdrom) via the built into mount option in the VMware Fusion menu.

Selecting 'Reinstall VMware Tools' to mount the installation files. Click for full size image.

Selecting ‘Reinstall VMware Tools’ to mount the installation files. Click for full size image.

Once that’s mounted, the machine is ready to install the tools on. However, there are a few other things to install just before installing these. First get the latest updates for apt-get with the update command.

sudo apt-get update

Now install the latest gcc, make, kernel headers and other important tools.

sudo apt-get install gcc make build-essential
sudo apt-get install linux-headers-$(uname -r)

In the above, everything can be put on one line, but I separated the linux-headers just for extra clarity. I can now via remote SSH on the local machine or directly into the virtual machine and run the following commands to install the VMware Tools.

sudo mkdir /mnttools
sudo mount /dev/cdrom /mnttools
tar xzvf /mnttools/VMwareTools-x.x.x-xxxx.tar.gz -C /tmp/
cd /tmp/vmware-tools-distrib/
sudo ./vmware-install.pl -d

Finish everything up with a good reboot.

sudo shutdown -r now

Now I have the VMware Tools installed and able to SSH remotely, giving me the ability to use the virtual machine as I would an actual hosted instance.

That Was Fun, Done With The Lenovo Carbon X1, Back to GSD!

Over the last couple of months I’ve been double laptoping it. I’ve had a Lenovo Carbon X1 with Windows 8 and Ubuntu dual boot configuration with 8GB RAM, 256GB SSD and i7 and I had a Mac Book Air (MBA) 8GB, 512GB SSD and i7 Proc. The MBA was my primary work machine with the Lenovo being a secondary machine that I was using to test and build Windows 8 Applications and for building native Linux services and related code work.

Windows 8 Critique

Simply, Windows 8 is one of the most broken operating systems I’ve used since Windows ME. Forget Vista, I consider it officially dethroned. Let me clarify what is and isn’t horrible about Windows 8 though. It isn’t that it technically is a bad operating system, it’s that the idea and approach that Microsoft has taken is inherently flawed at several key points.

First, having a desktop on a tablet, which is almost impossible except for all but the finest of finger pointing tablet users, is blindingly stupid. Just go into any place where there is a Windows 8 tablet user and watch them whacking away when they get into the desktop.

The Windows 8 desktop on a tablet is patently absurd for the vast majority of potential Windows 8 users.

However, the straight Metro Interface of Windows 8 (which Microsoft now calls the Windows 8 interface because of legal reasons) is magnificent for tablet usage. There are a few major things that need fixed: responsiveness related to connection state, update status and the availability of high quality applications. Once those things are fixed Windows 8 will be as competent as iOS or Android in the usability department. Until then, it’s a nice dream, with a small number of usable apps with a huge potential.

Now the desktop is the tried and true classic desktop of Windows. Thus, when you’re on a desktop machine or a laptop with a dedicated pointing device or touch screen the back and forth is fine. Matter of fact it is great! I find myself using the touch screen regularly to do a number of tasks, and hope to see its use increase more and more on a number of platforms (yo Apple, got game on this yet or not, OS-X can definitely use a touch interface).

Overall though, Windows 8 – unless you solely do Windows 8 Development, is not a reason to buy a Lenovo X1 Carbon.

Ubuntu Critique

Minus the touch screen, which Ubuntu has no clue what to do with except treat it like a pointer, this is how you see the real power and beauty of the Lenovo X1 Carbon. Ubuntu loads 2x faster and shuts down 2x faster than Windows 8. Comparable builds in IntelliJ, C, C++, Erlang and other compilers are regularly 1.2-3x faster than on Windows. The servers that one would build against, such as GlassFish (see this for my latest on setting up GlassFish & Java 7) are also routinely faster, more responsive and less prone to difficulty than in Windows.

One of the problems that is ongoing, is it is hard to move to Ubuntu unless you are doing dev. Using Adobe tools is a non-starter, best to stick to slow Windows or get real fast using OS-X. Again though, if it runs on Windows and Linux, I’d take a safe guess that the Linux versions will be faster, probably more stable, and all around it’ll likely work better over time. There is something to that whole unix way about building things. One other big booster for Ubuntu, is writing JavaScript, which I do regularly these days is a much better experience than on Windows. I use standard tools, that usually are available on Windows, but launching Sublime 2 or WebStorm is just faster, noticeably, on Linux versus Windows 8 (or whatever version really).

So overall, if you’re going to get a Linux machine, the Lenovo X1 Carbon is a prime choice. If not one of the best. If I understand correctly, there may even be some solid Linux software out there that would make the touch screen more usable too. So if you’re adventurous you may be able to solve that one single issue that I had with Linux running on the X1.

Would I Give Up My Mac for the X1?

This is easy, the answer is absurdly simple. However I did give up the Mac Book Air I had in parallel with the Lenovo for several months, as it belonged to Basho (which I’ve departed from).

Hell no!!!

Matter of fact, even though I’ve used the laptop extensively with Ubuntu and Windows 8, I’ve just bought a new Mac Book Pro Retina 15″ to do all of my work with Ubuntu, Windows 8 and OS-X. The solidness of the MBP is untouchable compared to the X1. The screen is better, the keyboard is more consistent and easier to type on, the ghost tracking of the track pad is non-existent on the air, versus the X1 Carbon. In this case, I’d even turned off the trackpad entirely on the X1 Carbon. Simply, the X1 Carbon just doesn’t measure up to the Mac Book Pro.

Other observations I’ve made about the two machines. The Mac Book Pro is far more solid, the construction is just not even comparable. The X1 feels solid but compared to the MBP it feels cheap and flimsy. Considering the hardware works flawlessly with the software on the MBP is also no competition. The Carbon regularly needed driver updates, things would flake out and I’d have to restart. This would be prevalent in windows or linux, it didn’t matter. Fortunately a restart would fix it, but none of these issues exist on the MBP, using either OS-X or running a VM with Windows 8 or Ubuntu.

Also, even though the MBP design is over a year old now, the i7, 16 GB RAM and 512 GB SSD makes the X1 Carbon seem like a morbidly out of date, slow and antiquated device even though it is actually a newer device!

So, would I give up my mac for the X1?