I’m Going to OSCON!

I’m very excited that I’m attending my first OSCON this year, compliments of The Cloudcast. On the April 25, 2016 episode, it was announced that they were giving away two Bronze Level passes to the conference. To win a pass, they asked to send in a personal story about community and why you wanted to attend OSCON. I wanted to share my story with a wider audience:

I’ve had an interest in Linux and coding since 6th grade and did some intermediate level coding in high school, college, and in my career. Finding help online to complete a coding project or successfully getting something like XWindows running in the late 90s was quite difficult. Similarly today, learning new, uncharted, and complex systems such as Kubernetes and Mesos is challenging but the community around it is fanatical and extremely helpful. I find that the community surrounding a technology is a key to it’s success and the success of the business. I’ve found myself gravitating to upcoming technologies that have deeply rooted and committed communities because those are the ones that usually become a mainstay in the industry.

One of my first experiences with such a community was the VMware community via Twitter in 2010. I started down the path of virtualization at my employer and my head was spinning. I happened to find that there was a vast community of virtualization evangelists that shared best practices and experiences and it was awesome to connect with hundreds of people who were there to help others. From there, I learned about local VMware user groups and wanted to participate in one to hear what others were doing and try to learn more. Living in Louisiana, which isn’t a major city/tech hub, there wasn’t a local group. I found tremendous value in the virtual community and I wanted the same experience of learning and sharing in person as well. I reached out to VMware and told them I wanted to start a local chapter of the VMUG. I started the first VMUG in Louisiana in Baton Rouge and a year later, due to it’s success, VMware asked me to form and run a second user group in New Orleans. I’ve been running the New Orleans group for the last 4 years and continue to enjoy learning about how organization’s environments are maturing with the maturation of the ecosystem.

As virtualization has solidified itself as a basic function of the datacenter, I’ve found myself following the maturation of the datacenter and it’s all pointing towards open source software. This has been very refreshing for me personally because it feels that there has been a reset in the datacenter as it begins to reinvent itself. Once virtualization became an anchor of the datacenter, VMware and other ecosystem partners moved up the stack to build upon that foundation. This has brought more end-user related products to the market but has left the infrastructure architect without many new tools with which to innovate. Containers will be a pillar of the next generation datacenter and it’s all fueled by open source software. I’m excited to see how the shift away from enterprise software vendors, which have been to  have rigid and archaic ways of designing software, changes over the next few years. The flexibility of OSS will give organizations a way to consume software through the community which is free and empowered to define how the software or application is best built. The community around these technologies will have a strong influence to guide it in the best direction but the biggest impact of the community is to be engaging with each other to guide newcomers and strengthen the established members.

I look forward to becoming part of this community and meeting new people next week.

9 Months at PernixData: Recapping My Best Career Decision

It’s hard to believe that my one year anniversary at PernixData is in 3 months. I have to say that coming here has been the best career decision I’ve ever made. I’m proud to be a part of the fastest growing software infrastructure company in history and surrounded by highly talented and passionate people. Having come from the customer side, specifically a medical practice, it’s interesting to be part of a software company and be a part of a startup.

PernixData is now a 3 year old company and I previously didn’t think I would ever join a startup. When I considered other opportunities at other young companies in the past, my family always cautioned me on having stability. I stayed at a 30 year old company for 10 years in the name of comfort and stability and I have some regrets about it. Louisiana isn’t busting at the seams with jobs in enterprise IT so the associated risk of joining a startup and losing my job one day weighed heavily on me. The security of an established organization was very comforting but I recognized that my growth was extremely limited due to the size of the organization and their plans for growth. Building the proverbial IT mansion was fun because I left the organization with a solid infrastructure but after the projects were over, the upkeep was minimal and the days became mundane. The decision to move on came down to fulfilling a desire to grow professionally by facing a new challenge outside of out of my comfort zone.

When going to a startup there can be a lot of risk and a lot of reward. Aside from believing in the architecture, what gave me comfort in joining a young company is it’s leaders. Satyam Vaghani and Poojan Kumar aren’t household names but they were both seasoned VMware alum that had a vision and brought along a team of world class developers that could execute that vision.

There’s risk with everything in life, but I opted to set aside my overly cautious feelings about job security and jumped right in because I only see PernixData continuing to grow.

The transition from Director of IT to Systems Engineer was exactly the change I was looking for because I was burnt out with operations. The career change also gave me the opportunity to explore existing, new, and upcoming technologies and understand how they relate to FVP. In my previous role, my experience with various hardware and software configurations were limited because there I felt learning about them wasn’t beneficial because they weren’t relevant to my job nor would we ever need to implement a solution like it. (Note: I recognize this was a terrible mindset to have and have since changed.)

As the Director of IT of a small company, I was responsible for maintaining the infrastructure and I was the manager of a desktop tech. The desktop guy was great at his job but I didn’t have a peer to collaborate with. That’s why starting a VMUG in Louisiana was important to me — I wanted a community of peers to share and learn with about virtualization.

The PernixData SE team is a great example of the peer community that I wanted to be part of. I’m surrounded by around 20 other engineers that have come from various backgrounds such as a fellow IT Director, virtualization admins, VMware instructor, and experienced SEs. Each of us has worked in different verticals and with different applications, hardware, and end users. This diversity allows each of us to bring our unique experiences and contribute them to the team and further develop a highly skilled and technical team. I’m also very proud that over 90% of our team are VMware vExperts.

What has been the most fun about working at PernixData is meeting people across the country and challenging them to re-think how they purchase storage and drive application performance. During the POC process, I love talking about technology with customers, learning about each company’s environments and challenges, and ultimately letting FVP speak for itself. In my opinion, being able to stand behind the product that you sell is ultimately what removes some of the challenges of being in sales and makes it enjoyable.

If you work for PernixData at HQ, there’s a lot of perks but as a remote employee I don’t benefit from but working at home is a great supplement. I don’t know how I could ever go back to working in an office, that’s for sure! Personally, the transition from office worker to teleworker hasn’t been difficult because I talk to quite a few customers everyday and always staying in touch with team members.

On a personal note, my wife is almost always home because she works at night as a registered nurse. For some, working from home while their spouse is there has presented challenges. This hasn’t been the case for us but YMMV! Another perk that I enjoy about working from home is being to take my kids to school occasionally and always being here when they get home from school. Once they’re home, they love to come into my office and keep me company for the rest of the day. It’s not always unicorns and rainbows though, I have to kick them out quite a bit!

Overall, I’m very pleased with how the last 9 months have turned out and always looking forward to the next day.

Shiny new vStuff — Speak at a VMUG NOW!

Having spent the last 3.5 years as a VMUG leader of two different VMUGs and spent time talking to over a dozen other leaders, one issue persists in the VMUG community: lack of customer participation. VMUG recognized this and implemented the Feed4ward program to, “encourage every interested member to share their knowledge at a VMUG local group meeting or User Conference”. Knowledge sharing is what everyone’s there for but most of the time people are nervous about public speaking, don’t think they know enough to discuss topics with others, or they think what they do isn’t that different or interesting. That can all changes now! 

With the release of vSphere 6 on March 12, everything is new to everyone. Not many people have downloaded it in their test/dev/lab environment and (hopefully) no one has deployed it in production yet! There are 11 vSphere ecosystem products that got updated and probably thousands of new features or enhancements to discuss. If you think just an “upgrading to ESXi 6” presentation will be boring, look at upgrading or starting to use one of the other supporting vSphere products such as vRealize Automation or Operations Manager. Maybe you’re a SMB and using or looking to use vSphere Data Protection or vSphere Replication. What was the upgrade or setup process like? How do you manage it? Did you ever have to recover from a backup or replica? Any gotchas? There’s plenty of opportunity now to get started giving back to your local VMUG community. If you want mentoring, look into the VMUG Feed4ward program!

The vSphere 6 documentation ca be found at: https://www.vmware.com/support/pubs/vsphere-esxi-vcenter-server-6-pubs.html.

Take this time to get out in front and start getting familiar with the new features and the associated documentation. Many organizations will look to upgrade once update 1 rolls around (I was in this crowd) which will probably be released in 6 months. Take the lead, become the expert, and be a staple in your local community.

The local VMUG leaders will probably already have a “What’s new in vSphere 6” slot carved out at the next meeting but if there’s a product feature or enhancement you like, love, or have always wanted to see, speak with them about adding a deep dive into that topic. It’s highly unlikely they’ll say no!

Better Documentation with Infrastructure as Code

Creating documentation isn’t fun. I’ve done my fair share in 10 years of administering systems. I’ve written documentation on AD, Exchange, router and switch configurations, VoIP system configuration and operations, and so on. As a one man shop that architected all the systems I ran, I was unsure what level of detail was required. What helped the most was having an outside resource that could review the documentation and try to fix a problem given the information I documented. Whatever question he had, that also had to be added.

What I’m not used to is taking over an infrastructure or application and being tasked to administer it. Even with decent documentation from the previous admin, you really don’t know the environment until you’ve had to fix a problem.

Recently, I tasked myself with taking over and update an internal application when the previous owner left. Because it was a small (but useful) tool, documentation was non-existent. To make the necessary updates to it, I had to spend plenty of time understanding how the application was structured. Once that I was done, I was ready to add my code and begin testing it.

Here’s the problem I was faced with: I don’t want to disrupt the application in production so I need a test environment but I don’t know everything I need to install to match the production application. Sure I could clone the VM, change the hostname and IP address, etc and hack at it that way. But there’s a better tool to tackle this with that will allow me to document the application and build the environment in a repeatable way. Enter configuration management and the concept of infrastructure as code.

Tools such as Puppet, Chef, and Ansible enable this ability. By implementing the concept of infrastructure as code, admins have the ability to provide useful documentation for systems and applications in the environment and also establish a mechanism to stand up additional application components or even provision new hardware. I chose to learn Ansible because I like the fact that the syntax is very simple (YAML), it uses SSH to communicate with the host, and is agentless.

From my investigation into the app, I know I need Ubuntu Linux, PHP, Apache, Postgresql, and Python. With a little command line-fu, I can find out which versions of the software I need and ensure that my configuration specifics those versions to be installed.

By implementing the concept of infrastructure as code, relevant and detailed documentation is provided for you, your team, and those that come after you.


Updating the firmware on a Micron p420m

Update: Micron released B218 in July 2015 to resolve a critical issue with command time outs. Micron highly recommends upgrading to this firmware release.

A common PCIe flash card for PernixData customers to use is the Micron P420m in their environment. It’s a very high performing and cost effective PCIe card and has a variety of applications.

Like all hardware devices, the p420m has firmware that occasionally needs to be updated. To perform the firmware update, we’re going to download the Micron rssdm utility (packaged with the ESXi drivers) on Micron’s site in the Support Pack for Linux and VMware package. As of January 2015, the support pack B145.03 from September 2014 is still current.

The first step to determining which firmware version the card is running is to install rssdm. Put the host into maintenance mode, copy the vib for your version of ESXi to the host, and run the esxcli software vib install -v command and reboot the host.

Once the host is back up, log in and execute the /opt/micron/bin/rssdm -L command to see the firmware version of the card.


As you can see, my card is running firmware version B2100600 and needs to be updated. At the time this article is posted, current firmware version is B2120500. We’re going to copy the new firmware to the host or shared datastore and perform the upgrade.

With the host in maintenance mode and the device removed from the FVP Cluster, copy the B145.03.00.ubi firmware image downloaded from the Micron Support Pack above to a location accessible by the host. The B145.03.00.ubi file will be in the Unified Image folder.

Then execute /opt/micron/bin/rssdm -T /path/to/file/B145.03.00.ubi -n 0


Once it’s complete, reboot the host.

When the host is back up, verify the new firmware is active.


Secure erase the drive by executing: /opt/micron/bin/rssdm -X -n 0 -p ffff

Screen Shot 2015-01-19 at 10.40.10 PM

Note: For those interested, -X is to perform the secure erase, -n is to specify the drive ID, and -p is for password (default is ffff)

After the secure erase is complete, remove the host out of maintenance mode and add the device back to the FVP Cluster.

Intro to Git and the Rise of Social Coding Presentation

Over the Christmas season, I participated in social coding community event called Commitmas. The objective was to learn git and try to use it every day for 12 days. As I wrote in another blog post, Rockin Around the Commitmas Tree, this was a great opportunity to jump start learning to use it with others in the community.

Since then I’ve continued using git on a regular basis for various side projects. I wanted the opportunity to share with the PernixData SE team a little bit about what I’ve learned and why it’s important for us to learn and use git. I feel that learning git is a crucial foundational skill for building and managing next generation datacenters and developing applications. I also like git for daily use to manage versions of documents or presentations.

Putting together this presentation was a bit special for me for two reasons. First, it’s the first technical presentation that I created that wasn’t related to my core competency, VMware vSphere. This stands out to me because it’s representative of my personal journey to be an early adopter of these new technologies. The configuration management/container/PaaS space is still in a very early phase and it reminds me of stories I’ve heard about having to perform vMotion on the command line. That’s always sound so old school to me because I started using vSphere in 2009 when 4.0 was released and it was all well established. I often wonder what this space will look in a few years and will it take less time to mature.

It’s also the first time that I’ve used reveal.js to create a presentation. I’ve had the chance to see a few presentations with done with reveal.js and really like the elegance and simplicity of it. As a technologist, I like the geekiness of creating my presentation with HTML.

Git: Syncing a Fork In It

On the second day of Commitmas, I had a new hurdle to jump — I forked @mjbrender’s repo, submitted a PR, and he accepted my changes as well as changes from @joshcoen and now my fork is outdated. How do I update my fork and local repo with the changes?

To rectify this, we’re going to leverage two git commands: git remote add upstream and git fetch upstream command.

With git remote add upstream, we’re going to specify the repository that we forked. In my case, I did:

git remote add upstream https://github.com/mjbrender/12-days-of-commitmas

A git remote -v verifies that we added the upstream source:

Screen Shot 2014-12-22 at 10.12.15 PM

Then we perform git fetch upstream to copy the original repository (the one we forked) to our local repository (on our dev box):

Screen Shot 2014-12-22 at 10.12.23 PM

Now we just fetched the upstream/master branch/repo and needs to be merged into our master branch. We can switch to our master branch with git checkout master.

Finally, git merge upstream/master to merge upstream/master to our master branch.

Screen Shot 2014-12-22 at 10.12.55 PM

The final stretch! Our local repository is now a clone of the original repository again but we need to push our changes back to our repository on GItHub.

Screen Shot 2014-12-22 at 10.28.16 PM

Done! Join in the Commitmas conversation on Twitter using the #vBrownBag hashtag! Happy committing!

Rockin’ Around the Commitmas Tree


My good friend Matthew Brender started a fun project for the holiday season called 12 Days of Commitmas to encourage everyone to continue to grow over the holiday season by learning or expanding their knowledge with git and coding. These skills will become increasingly prevalent and necessary in 2015 as more sysadmins and organizations implement version control to manage code for configuration management tools like Ansible.

When looking at the skill levels that Matt created, I found myself between beginner and intermediate. In the spirit of code contribution and learning git, and playing with Markdown, I submitted my first pull request on GitHub to expand the skill levels.

My Goal for Commitmas

Because my coding skills are relatively weak (improving them is a 2015 goal), I will contribute documentation to a project on GitHub. Internally on the SE team at PernixData, there are some PowerShell scripts that I will create repositories for in GitLab. I haven’t picked a project on GitHub yet to contribute to but I’m considering something related to Ansible. This will give me a chance to explore the different Ansible projects on GitHub and learn/do more with Markdown. I’m also going to push all of my commits using the CLI as I currently lean heavily on the GUI. I know there’s nothing wrong with using the GUI but I believe that true mastery comes from using the CLI when applicable.

Join in the Commitmas conversation on Twitter using the #vBrownBag hashtag! Happy committing!

Use Case for Invalidating PernixData Read Cache

When sizing an acceleration resource for PernixData FVP, we look to recommend a resource that can contain the working set of data for the VMs using that resource. The majority of the time this isn’t a problem. However, we recently came across a use case where the working set of the VM was so large that the VM used the whole flash device and began garbage collection after just 3 days. Let’s take a closer look at what’s going on and how we found a feasible solution for the customer.
The workload is a batch processing application that runs once a day and churns through a lot of data during it’s processing time. Before implementing FVP, this was taking approximately 2.25 hours. The first day that it ran after installing FVP, the processing time was reduced a noticeable amount as FVP accelerates every write operation in write back but caches blocks for read acceleration as the VM requests them. By the 2nd or 3rd day, batch processing time was nearly reduced by 50% and customer was thrilled. But the 4th day, processing time was creeping back up to around 2.25 hours. It turned out that even with an 800 GB SSD, the drive filled up and was affected by frequent write amplification and garbage collection to take on newly read blocks and service the writes. So what about stringing some SSDs together in RAID0?
This is a very common question that we get so let’s investigate SSD and RAID a bit.
First, it’s important to understand that SSDs don’t fail like HDD. As SSDs near failure, they might continue to service IO but at an extremely high and invariable latency whereas we typically see consistent performance. In turn, a single SSD that is failing will drag down performance of the RAID array because the array can only performa at the latency and throughput of the slowest SSDs.
As a side note, FVP has adaptive resource management. As soon as FVP determines that going to a given flash device is slower than simply using the backend SAN, it will correctly deactivate said flash device from the FVP cluster. This is technology that is built from ground up to deal with the failure characteristics of SSDs (i..e good ssd vs slow SSD vs failed SSD). RAID was never built to make a distinction between a slow SSD and a failed SSD. Being highly available with an impractical-to-use flash device is pointless — that is actually worse than being not available.
The customer is also using blade servers so a larger capacity PCIe card isn’t an option. So the customer has the largest available capacity in SSD format. Currently the customer needed a solution to consistently achieve the reduction in batch processing time.
What can we do?
PernixData FVP ships with some very robust PowerShell cmdlets that allows us to manage the environment and automate all the things. We first considered using PowerShell to remove the VM from the FVP cluster every few days to force the cache to drop but that would also lose all the VM performance graphs. Instead, our solution was to use PowerShell to blacklist the VM every 3 days. The customer is happy with this solution because even on the first run with batch processing and FVP, run time was still better than straight to the array.
What does this look like?
#Setup some variables
$fvp_server = "fvpservername"
$fvp_username = "domain\username"
#TODO: This isn't very scalable for large environment. I know there's a better way.
$vms = "vm01", "vm02"
#A file we're going to use to store the password in an encrypted format
$passwordfile = "c:\path\fvp_enc.txt"

import-module prnxcli

# Authentication and connection strings removed for brevity. 

#Loop through the list of VMs and set each to be blacklisted, wait 30 seconds then add them back to cluster in WB
foreach ($vm in $vms)
     write-host "Blacklisting $vm to invalidate cache"
     Set-PrnxProperty $vm -Name cachePolicy –value 1
     Start-Sleep -seconds 10
     write-host “Adding $vm back to FVP cluster in write back"
     Set-PrnxProperty $vm -Name cachePolicy -value 3
     Set-PrnxAccelerationPolicy -WaitTimeSeconds 30 -name $vm -WB -NumWBPeers 1 -NumWBExternalPeers 0


This scenario isn’t very typical but with large data sets, sometimes you need to get creative!

Clone this repo on GitHub to get the full script or fork and make it fancy! https://github.com/bdwill/prnx-invalidate-cache

Goals for VMworld 2014

VMworld 2014 will be my second time at VMworld and because the conference goes by so fast, it’s important to make a list of goals to achieve and work to stick to it. You don’t want to look back on your time at VMworld and say, “I wish…”

My first VMworld was in 2011 and was an amazing experience. I had been using Twitter and been involved with the community for about 2 years and meeting many people I followed and became friends with was a great experience. I had also been a VMUG leader for 2 years and enjoyed the opportunity to meet and mingle with other VMUG leaders.

VMworld 2012 would have also been another first for me as I was honored as a vExpert 2012 and was really looking forward to participating in the vExpert party. However, a category 3 hurricane was bearing down on New Orleans and I didn’t want to leave my family. I hope to join the current ranks of vExpert for Q3 2014 and participate next year.

Now that I’m at PernixData, my time at the conference will be spent completely differently than my first time. As a vendor, you don’t typically get to attend sessions. A fair amount of my time will be spent at the PernixData booth (#1017) demonstrating FVP. So the role as a vendor is going to be a completely different but still exciting. PernixData has built an incredible product and is already changing how datacenter are designed. Not many startups can say that after selling for 1 year. I’m surrounded by such an incredibly talented team and looking forward to spending time with them this week.

So what do my goals for VMworld 2014 look like?
Evangelize PernixData FVP. My primary reason for being at VMworld is to share PernixData FVP with the world so without a doubt, that’s goal #1. PernixData is an incredible product that I really love so it’s very easy and fun. I’ll be at the PernixData booth during the week so stop by and say hi.
Network with VMUG leaders. VMUG has grown tremendously since 2011 and I’m looking forward to meeting new leaders and continuing to learn how to improve the New Orleans VMUG.
Attend vBrownBag sessions. Without the ability to attend vmworld sessions, I’m looking for some avenue to learn while surrounded by so many smart and talented people. The vBrownBag crew has lined up some great speakers for the week and is sure to be a place I spent a lot of personal time. I’ve also recently co-hosted a couple of weekly sessions so it will be great to meet everyone in person.
Network with the VMware community. PernixPros, PernixPrimes, other vExperts, and everyone else in between: I want to meet you. Everyone has a story to tell so I look forward to growing personal and professional relationships.
Blog. I’ve been wanting to do this for years and never set the time aside to do it. With so many new announcements from VMware and partners, it’s the perfect opportunity to kick things off.
Time will fly so make a list of your goals and stick to them!