Some vim goodness ...

As you would expect of a long term linux user, I am familiar with vim. Most who use it know that a PhD is needed to actually drive it with any amount of competency.

For this reason I am currently using Sublime Text 3 (on Mac), which by the way is awesome (http://www.sublimetext.com/3).

The team I am currently working with are highly skilled and are big users of vim so I have made it my mission to start the transition back to it and thought I would share some of the learnings that I have recently made.

1. Working with split screens

When working with things like Puppet and Ansible, it is often very useful to have 2 files open at once, either to copy something that was done before or when tracing down an issue.

So here is the scenario, I have file1 and file2 and I want to get them on the screen with a vertical split.

Firstly we open file1:

vim file1

Then split the screen vertically:

ESC + :vs

This will split the screen vertically, your cursour will be on the left hand side.

To move between the split you just use:

CTRL + w then L or R arrow

You can do the ":vs" more than once so can have more than 1 split going in a window, moving around it is the same.

To close down a split screen simply exit vim using something like:

ESC + :q (or :wq if you want to write changes)

2. Multi line changes

This is a very powerful action and one that I rely on every day in Sublime Text. This method uses vim's visual blocking or highlighting.

Here are a couple of examples on how do use this:

Indent a block of text 4 spaces

place the cursor on the first character of the first line
ctl+v (vertical select)
arrow down rows
shift+i
Press space 4 times
esc (apply to all lines)

In action: https://www.youtube.com/watch?v=o4ZvVZDyBV4

Amendment:
It has also been bought to my attention, if you want to just do a single indent (ie 2 spaces) you can use `>` (Thanks MichaelD)

Delete 4 spaces from the beginning of a block of text

ctl+v (vertical select)
use the arrows to highlight a block
press x (x usually is used to delete a single character)

In action: https://www.youtube.com/watch?v=fQrX4OYI2_w

3. vim customisation

There have been a number of things whilst using Sublime Text that I have really gotten use to, spaces used in tabs, line numbers etc etc.

vim has some very powerful customisations, this is controlled by a file called .vimrc and usually lives in the root of a users home directory ie ~/.vimrc

Similar to ruby's bundler vim has Vundle, this allows you to nicely manage plugins for vim.

https://github.com/gmarik/Vundle.vim

Here is a really good article on how to set it up

https://www.digitalocean.com/community/tutorials/how-to-use-vundle-to-manage-vim-plugins-on-a-linux-vps

Once you have vundle installed it is a simple matter of installing extra plugins for use. A good start is a colour theme, I am a fan of the colour scheme w0ng/vim-hybrid.

To install it you do:

vim then :PluginInstall w0ng/vim-hybrid

Then you add it to your ~/.vimrc.

Here is what my current file looks like:

Bundle 'w0ng/vim-hybrid'
color hybrid
syntax on
set number
set tabstop=2 softtabstop=0 expandtab shiftwidth=2

Hybrid is a really nice colour scheme that has great syntax highlighting for things like ruby, puppet and ansible. It is also really good when using vimdiff.

You can find a heap of plugins to install here

http://vim-scripts.org/vim/scripts.html

I also like to have the line numbers showing by default, this is done with `set number`. I also like to have softtabs set to 2 (ie tabs are achieved with spaces).

Given the complexity of vim you can do some amazing things with this customisation, many I have yet to discover but will share once I do.

Until next time.

Parsing JSON with Ansible ...

Recently, I have been doing a heap of work around the automation
and provisioning of resources with AWS cloud services. This entails working
frequently with the AWS API.

A useful filter we have been using in Ansible is taking output from a shell
action and turning it into something we can consume via variables.

An example playbook is below:

- shell: |
    export AWS_ACCESS_KEY_ID=$AWS_ACCESS_KEY
    export AWS_SECRET_ACCESS_KEY=$AWS_SECRET_KEY
    lib/ec2.py --list --refresh-cache
  register: output
- set_fact:
    ec2_output: "{{ output.stdout|from_json }}"
- shell: "rm addresses.txt"
- shell: "echo '{{ item.value['ec2_private_ip_address'] }}' >> addresses.txt"
  when: item.value["ec2_tag_Name"] is defined and item.value["ec2_tag_Name"] == "deis-{{ environment_unique_id }}"
  with_dict: ec2_output._meta.hostvars
- shell: "cat addresses.txt"
  register: output
- add_host:
    hostname: "{{ output.stdout_lines.0 }}"
    groupname: "deis_node"
- debug: var={{ groups['deis_node'].0 }}

This example uses an Ansible plugin script ec2.py to retrieve a list of instances
running in an AWS account. It then filters out the ones that we want via the
`ec2_tag_Name` key and adds the private IP address to a file.

After running the following shell action, `output` holds a heap of information
about what the commands did and the output from stdout and stderr (if there was
any.)

- shell: |
    export AWS_ACCESS_KEY_ID=$AWS_ACCESS_KEY
    export AWS_SECRET_ACCESS_KEY=$AWS_SECRET_KEY
    lib/ec2.py --list --refresh-cache
  register: output

The Ansible output for the registered variable output looks something like this:

TASK: [debug var=output] ******************************************************
ok: [localhost] => {
    "output": {
        "changed": true,
        "cmd": "export AWS_ACCESS_KEY_ID=$AWS_ACCESS_KEY \nexport AWS_SECRET_ACCESS_KEY=$AWS_SECRET_KEY \nlib/ec2.py --list --refresh-cache",
        "delta": "0:00:05.268682",
        "end": "2014-10-05 11:43:05.235594",
        "invocation": {
            "module_args": "export AWS_ACCESS_KEY_ID=$AWS_ACCESS_KEY \nexport AWS_SECRET_ACCESS_KEY=$AWS_SECRET_KEY \nlib/ec2.py --list --refresh-cache",
            "module_name": "shell"
        },
        "rc": 0,
        "start": "2013-10-05 11:42:59.966912",
        "stderr": "",
        "stdout": "<suppressed to keep it short>",
        "stdout_lines": [
            "{",
            "  \"_meta\": {",
            "    \"hostvars\": {",
            "      \"xx.xxx.xxx.xxx\": {",
            "        \"ec2__in_monitoring_element\": false, ",
            "        \"ec2_ami_launch_index\": \"0\", ",
            "        \"ec2_architecture\": \"x86_64\", ",
            "        \"ec2_client_token\": \"xxxxxxxx-xxxx-xxxx-xxxx-c55977fc0029_us-east-1a_1\", ",
            "        \"ec2_dns_name\": \"ec2-xx-xxx-xxx-xxx.compute-1.amazonaws.com\", ",
            "        \"ec2_ebs_optimized\": false, ",
            "        \"ec2_eventsSet\": \"\", ",
            "        \"ec2_group_name\": \"\", ",
            "        \"ec2_hypervisor\": \"xen\", ",
            "        \"ec2_id\": \"i-xxxxxx\", ",
            "        \"ec2_image_id\": \"ami-xxxxxx\", ",
            "        \"ec2_instance_profile\": \"\", ",
            "        \"ec2_instance_type\": \"m1.large\", ",
            "        \"ec2_ip_address\": \"xx.xxx.xxx.xxx\", ",
            "        \"ec2_item\": \"\", ",
            "        \"ec2_kernel\": \"aki-xxxxxxxx\", ",
            "        \"ec2_key_name\": \"ambari\", ",
            "        \"ec2_launch_time\": \"2013-10-01T04:53:00.000Z\", ",
            "        \"ec2_monitored\": true, ",
            "        \"ec2_monitoring\": \"\", ",
            "        \"ec2_monitoring_state\": \"enabled\", ",
            "        \"ec2_persistent\": false, ",
            "        \"ec2_placement\": \"us-east-1a\", ",
            "        \"ec2_platform\": \"\", ",
            "        \"ec2_previous_state\": \"\", ",
            "        \"ec2_previous_state_code\": 0, ",
            "        \"ec2_private_dns_name\": \"ip-xx-xx-x-xxx.ec2.internal\", ",
            "        \"ec2_private_ip_address\": \"xx.xx.x.xxx\", ",
            "        \"ec2_public_dns_name\": \"ec2-xx-xxx-xxx-xxx.compute-1.amazonaws.com\", ",
            "        \"ec2_ramdisk\": \"\", ",
            "        \"ec2_reason\": \"\", ",
            "        \"ec2_region\": \"us-east-1\", ",
            "        \"ec2_requester_id\": \"\", ",
            "        \"ec2_root_device_name\": \"/dev/sda1\", ",
            "        \"ec2_root_device_type\": \"ebs\", ",
            "        \"ec2_security_group_ids\": \"sg-xxxxxxxx\", ",
            "        \"ec2_security_group_names\": \"ambari\", ",
            "        \"ec2_sourceDestCheck\": \"true\", ",
            "        \"ec2_spot_instance_request_id\": \"\", ",
            "        \"ec2_state\": \"running\", ",
            "        \"ec2_state_code\": 16, ",
            "        \"ec2_state_reason\": \"\", ",
            "        \"ec2_subnet_id\": \"subnet-xxxxxxxx\", ",
            "        \"ec2_tag_Name\": \"hdpmaster1\", ",
            "        \"ec2_tag_aws_autoscaling_groupName\": \"ambari-LargeClusterGroup-148W3OVR9LSSE\", ",
            "        \"ec2_tag_aws_cloudformation_logical-id\": \"LargeClusterGroup\", ",
            "        \"ec2_tag_aws_cloudformation_stack-id\": \"arn:aws:cloudformation:us-east-1:167609138788:stack/ambari/xxxxxxxx-xxxx-xxxx-xxxx-50fa526be49c\", ",
            "        \"ec2_tag_aws_cloudformation_stack-name\": \"ambari\", ",
            "        \"ec2_tag_long_hostname\": \"hdpmaster1.domain.com\", ",
            "        \"ec2_virtualization_type\": \"paravirtual\", ",
            "        \"ec2_vpc_id\": \"vpc-xxxxxxxx\"",
            "      }, ",
            "}"
        ]
    }
}

As you can see stdout and stdout_lines aren't in a great format to do much with,
here enters from_json:

- set_fact:
    ec2_output: "{{ output.stdout|from_json }}"

This then allows us to do the following and `ec2_output` is now in a standard
JSON format that Ansible can then use to iterate over like so:

- shell: "echo '{{ item.value['ec2_private_ip_address'] }}' >> addresses.txt"
  when: item.value["ec2_tag_Name"] is defined and item.value["ec2_tag_Name"] == "deis-{{ environment_unique_id }}"
  with_dict: ec2_output._meta.hostvars

This Ansible task creates a file that contains all of the private IP addresses
of ec2 instances that match the `ec2_tag_Name` criteria.

We could then take the first of these IP addresses and use it later in our playbook:

- shell: "cat addresses.txt"
  register: output
- debug: var={{ output.stdout_lines.0 }}

This becomes a very powerful pattern when you mix in RESTful APIs, calls to them
and processing output from them.

The importance of the inspect and adaption cycle

At my work, we pride ourselves on our ability as an development team
to continually improve not only our engineering practices but also our planning,
testing, and communication.

Recently we performed an operation on one of our primary databases. This
involved creating a new slave database for our backups and other read only
services, promoting the current slave to be the primary as it had more resources
available to it, then decommission the old master.

This process was given a two hour window to be completed. The team involved
performed this work smoothly in just over an hour.

Post-work, we met as a team to perform a short retrospective; a common "meeting"
in the Agile/Scrum process which gives the team the chance to review what was
done, what worked, and what could be improved next time.

Here was the outcome from our retrospective:

Things we did well

• Planning and documentation processes have improved immensely over the last couple of maintenance windows
• Having the team onsite yet again proved to be superior to remote - communications and responsiveness to issues was great

Things we can improve

• Obtaining a maintenance window was problematic. Given my work is a global business, finding a convenient and appropriate time for a maintenance window has always proved challenging
• Generate and document success test cases that are going to be used to verify the system is ok in advance. This will help prevent us tripping over application bugs on the day, which could hamper the successful completion of maintenance

Actions

• Develop and maintain a maintenance window calendar that the business has access to so that if a window is needed a call can be made without involving engineering
• All maintenance windows should have a dress rehearsal done before a proposed window is requested. This is to ensure the team are across the process that needs to be done and more accurate timings are defined for the outage window

As agile practitioners its vital that we continually look at ways to improve on what was done so that next time it can be done better.

Day 21

 As I mentioned yesterday because I was so sore from the ride we decided to go with the SAG (Support and gear) wagon.

We woke early for breakfast as the SAG wagon was going to set out at 9:30am, this is what we had right outside our window.

This is a team car from one of the teams in Le Tour de France. Apparently (according to Jan) a lot of the he teams stay in Lourdes as it is close to the common mountain stage climbs each year.

Our job today was to support the group of riders that were doing the longest ride, 90kms.

Here are a couple of snaps from the van window.

Not too bad considering we were doing about 100 and the glass for glossy and tinted.

Now there were 2 climbs to be done on this ride, The first was

Here was the view

This place had 2 cafés on it and a heap of horses and sheep.

This guy was as stubborn as, and this is the van we were travelling in all day.

Kelly enjoying the wild life.

A panoramic!

I also had a coffee while we waited for the riders to crest the top.

Ah here some are now

Some more beautiful shots

The second climb was, Matt look an eagle!!

This also had a cafe located right on the top of the mountain. The reason why these guys stay in business is the cycling traffic they get most of the year round.

The way there was stunning

Some Tom foolery on the way

Some nice shots too

To get to this summit you had to got through here

 

There was another one that was long and was completely dark so the riders had to essentially feel their way as they didn't have any lights on.

Found this in the cafe on the summit.

The guy who won the 1926 Le Tour de France. In those days you had no support, it was much longer and you had to fix your own bike. If you broke a front fork you had to go to a blacksmith and repair it yourself, the blacksmith was not allowed to help otherwise you got disqualified. This guy won it an hour ahead of his rivals, now that is tough stuff. What was also interesting is that he owned a cafe on this very mountain when Jan (our driver) was growing up and was considered a French legend.

Here are some pictures from the top of this climb.

On the way down the worst thing happened, we came across an accident, someone on the decent.

He had over cooked the corner and "supermaned into the side hill", he was in a ditch on the side of the road made as comfortable as he could be by his friends. This picture was us waiting for a helicopter (that eventually didn't arrive) to air lift him out. It was quite scared as pretty much everyone didn't speak French so it was a real struggle until Jan (our driver) came to the rescue (incidentally he speaks 4 languages, legend).

We found out later the guy had 5 broken ribs, broken collar bone, broken other shoulder and 3 fractured vertebra. So a word of caution to all cyclist please be careful and mindful when decending, cycling can be a very dangerous sport (I am looking at you Andi and only because we care *grin*).

We were dropped off by Jan at the bottom of the Hautacam, the final climb for the day for Le Tour.

Here is what the crowd was like

And this went for miles. 

Kelly got a cracking shot of the leader of the General Classifications Nibali. He is the one in the the Yellow jersey behind his 2 team mates in aqua.

A couple more for good measure.

It was home again to the hotel.

In an attempt to make sure I was going to be ready for tomorrow's ride Jan offered to give me a once over as he bought his massage table with him, I gratefully accepted and went to his room at 7:30pm, it was awesome. We chatted about his experiences with the pro cycling teams, very interesting.

We then went out with the tour guides for dinner, another night shot of Lourdes.

This is what I had, freshly made, I saw the base being made by the cook, very very tasty.

Dinner seems to be getting later and later hear, mostly because we get back late but also because it is light until 10pm.

Looking forward to a ride tomorrow.

Day 22

Today was a ride from Lourdes to Luz Sauveur with the hope that we could tackle Col du Tourmalet. It was an amazing journey, some pictures below.

We arrived. The ride was very wet but very enjoyable. Unfortunately the weather didn't fine up so we were not able to give this monster of a mountain a go, a good excuse to come back next time. :)

Stopped for a well earned coffee ... oh look today's selfie.

We had a look around this beautiful little town which in the winter is a ski area.

Just over the bridge above is a bakery where I had the best almond croissant I have ever eaten, it was filled with custard to boot!!

This creek/river runs through the middle of the town.

We strolled over the bridge and found this beautiful church on a hill with an incredible view.

We went back into town and found this castle which also had a church inside it.

Oh look another castle on the hill.

This gothic catherderal (wow I am so travelled now *grin*), it in Lourdes.

Look another castle, just can't seem to get enough of them.

More pictures of this interesting church. 

So do anyone know why Lourdes is so famous? ... I didn't.

In 1858 there was a holy event, a Meridian apparition. This is described by Wikipedia (thanks Greg, it knows everything! *grin*) as a super material appearance by Blessed Virgin Mary. Apparently the site of this appearance was the Grotte.

In here there is a spring and apparently a local (or traveller not sure) was told that the waters from the spring would heal him. Now Lourdes is a bit of a contradiction. When we first came into town we thought we had ended up in a Thailand market street, junk from start to end selling everything you can think of religeous. Crosses, statues, badges etc, here is an example.

I mean how many statues a year can you sell!! But the truth is this place is packed, day and night all year round. Lourdes has 5 million visitors a year which is roughly 14,000 a day. The town never stops.

It had been a long day so we went back to the restaurant we we to the first night in Lourdes, and I had the same dish.

Tomorrow is a travelling day.

Until then.