The Flatiron School

The Flatiron School is a 12 week intensive program designed to turn you into a web developer.

This is a guest post by Dinshaw Gobhai of Constant Contact based on a talk he gave on campus

To a developer, operations can seem overwhelming, mysterious, and sometimes just boring. ‘DevOps’ is everywhere, but what people are referring to? Is it tools? Is it culture? Is it something one can learn?

The answer is yes.

How did we get into this mess?

At the dawn of the computer age, there was no distinction between dev and ops. If you developed, you operated. You mounted the tapes, you flipped the switches on the front panel, you rebooted when things crashed […]. Dev and ops started to separate in the ’60s, when programmer/analysts dumped boxes of punch cards into readers, and “computer operators” […] scurried around mounting tapes […]. The operators pulled printouts from line printers and shoved them in labeled cubbyholes, where you got your output filed under your last name […]. The arrival of minicomputers in the 1970s and PCs in the ’80s broke down the wall between mainframe operators and users, leading to the system and network administrators of the 1980s and ’90s. That was the birth of modern “IT operations” culture[…].

What is DevOps?, Mike-Loukides

Where did DevOps come from?

2007 Ghent, Belgium, Patrick Dubois was making an effort to work in every capacity that an IT contractor could. Professional experience in both development and operations gave him a clear view of a conflict of interests within all established IT infrastructure:

Development’s job is to introduce change. Operation’s job is to keep the site stable and fast.

Dubois is credited with informally inciting the DevOps community by initiating conversation around this paradox.


Slide from Flickr: Dev vs Ops

Fast-forward to 2009 at O’riliey’s Velocity Conf and the now famous talk: 10+ Deploys Per Day: Dev and Ops Cooperation at Flickr. John Allspaw (operations) and Paul Hammond (development) were the first to publicly call out the rift between dev and ops. We, as an industry, had created two job silos with contradictory goals. An ‘us vs them’ mentality had created mistrust and disrespect between dev and ops, and it was self-perpetuating: the better each side did at their job, the worse they made things for the other. Allspaw & Hammond described what they needed to solve this problem with a single slide:

Sadly, Patrick Dubois was not able to attend Velocity Conf ‘09. But while lamenting his situation over social media, friends suggested that he hold his own conf. So he did, and DevOps Days 2009 was born. The conference included Introducitng Kanban in Operations, Continuous integration pipelines and deployment, and Building agile infrastructure with Puppet, all familiar concepts today, but ground breaking at the time. DevOps Days was a hot topic on Twitter where the ‘Days’ was dropped, the term coined, and the movement galvanized.

DevOps stood out from the best-practices of the time because, instead of being another vendor-to-the-enterprise-set’s million dollar solution, this was from practitioners, for practitioners. DevOps was a grassroots movement against legacy tools and culture. It was not a spec, a product, a standard, or a job title. It was a lot of smart people collaborating on a complex set of common problems.


What’s the problem?

The problem that DevOps addresses, to sum it up, is the hand-off from Dev to Ops. When developers ‘throw’ their code over the proverbial wall to Ops, and it breaks, the cliché response is ‘Well, it worked on my machine… it must be your fault’. When software works in one place (dev environment) and not in another (test, stage, prod, etc) people get defensive, tensions rise, and cooperation becomes harder just when we need it most.

Our software gets more complex every day. The more complex our software gets, the more fragility creeps into our systems and, to quote Gene Kim:

"Critical systems cannot be fragile." Gene Kim, a DevOps evangelist, gives his Why We Need DevOps Now talk all over the place. If you haven’t seen it live, watch a recording. It is inspiring and informative.

What’s the solution?

At the heart of the DevOps ideal is an acknowledgment that Dev and Ops can no longer be separate responsibilities. Awareness of how code travels from development to production and how it will be deployed must be a part of the process from the start. Developers must understand their production systems, and inevitably own the uptime of the code that they write.

Our healing comes in two forms: Culture, and Tools.

Culture

Developers will need to learn about their production environment. This learning will likely come from someone in operations. Healing the divide between dev and ops, and fostering communication in the archetypal IT organization is the goal of DevOps culture.

Tools

There are many tools that can be labeled ‘DevOps’, and many more arrive each day. Victory is not choosing the perfect tool, but implementing common tool-chains across teams. When everyone is using the same tools we achieve transparency across teams. This fosters communication and appreciation of what our colleagues are doing, and ultimately (hopefully!) respect. It also enables us to work with each other with ease. Version control (we use Git), CI (we use Jenkins), and infrastructure automation (we use Puppet) are all good examples of tools that should be common across an organization.

Getting Started

TDD (skip to Automation if you already do TDD): You can’t automate any part of your code promotion until you can programmatically verify your work.

Automation: Starting down the read of automation can be daunting. Start with something small. Here are a couple suggestions for noobs:

  1. Make your editor clean trailing whitespace on save
  2. Create a .dotfiles repo
  3. Use Puppet or Chef to automate the configuration of your developer machine
  4. Use Vagrant or Docker to simulate your production env on your local machine

Have a cup of coffee with your closest ops person. Make friends, you are on the same team! Ask them about Puppet or Chef or what they do all day.

Much like everything else in the world, if this seems hard at first it will get easier!

The @FlatironSchool team behind opengovtimeline.com, a navigable historical timeline of  #NYCOpenData -via @n0n3ck

The @FlatironSchool team behind opengovtimeline.com, a navigable historical timeline of #NYCOpenData -via @n0n3ck

The @FlatironSchool team behind opengovtimeline.com, a navigable historical timeline of  #NYCOpenData -via @n0n3ck

The @FlatironSchool team behind opengovtimeline.com, a navigable historical timeline of #NYCOpenData -via @n0n3ck

Porting SQLite3 Data to PostgreSQL Database in Rails 4

The following is a guest post by Joe O’Conor and originally appeared on his blog. Joe is currently in the Ruby-003 class at The Flatiron School. You can follow him on Twitter here.

This past week I’ve been building a database with almost 1,000,000 records across five tables. So far, it has been just fine to use the Rails default, SQLite3, to manage it. That’s no longer the case, as I need to start thinking about the deployment environment for my application, and I have also been running into some issues with my SQLite3 database locking as I try to add records. Thus, I have ported my data over to PostgreSQL. The process is straightforward, but I did run into a few hangups and did not find a great guide to cover my situation. After some struggling, I got my new PostgreSQL database up and running with all of my data. Here’s how I did it.

First, I had to create a dumpfile from my sqlite3 database. I had been doing this already as quick way to back up my data. I read that it might be more convenient to dump it into a different format, like CSV, that is better understood by other databases like PostgreSQL, but I used the normal SQLite3 command and found that the manual adjustments from there were not cumbersome.

This is the path to my database: db/development.sqlite3. The .dump command breaks down your database into SQL statements, including everything you need for your schema. Then you indicate where you want to direct the dump using >. I created my dumpfile in the same directory and added a timestamp. Since the dump gives you SQL statements, the file extension should be .sql.

This file is sufficient to reseed another SQLite database, but PostgreSQL databases have some differences in syntax and datatypes. At first, I looked around to find a script or an application to manage these changes for me. My advice is: don’t bother. Finding one is hard enough, then you have to learn how to use it. A simple find/replace in your dumpfile will solve it just fine. This is what I had to change:

I only had to update this last one because I had entries in “description” columns that were too long. According to this very helpful post on StackOverflow, you may also need to change your boolean values into a type that is compatible with PostgreSQL. My booleans were already formatted correctly. For reference, here’s the list from the PostgreSQL documentation:

“Leading or trailing whitespace is ignored, and case does not matter. The key words TRUE and FALSE are the preferred (SQL-compliant) usage.” docs

The next step is to take out the SQLite-specific statements. At the top of my dumpfile, I had PRAGMA foreign_keys=OFF;. This tells the database to not enforce foreign key constraints, but PostgreSQL handles these differently. I took out the statement with no issue.

At the bottom of my dumpfile, I had a statement like this for each of my tables:

INSERT INTO "sqlite_sequence" VALUES('ks_users', 227393);

These tell SQLite where to set the next value for the autoincrementing primary keys of my tables, which in the example isks_users. PostgreSQL does the same thing, but in a different way, using sequences, which are special database objects designed for this purpose. We’ve already told our PostgreSQL database to use sequences when we changed our primary keys toSERIAL instead of INTEGER with AUTOINCREMENT, so the next step is just to take out all references tosqlite_sequence tables. (If you need to add new records to your database after the conversion, you’ll need to make one other change that we will talk about a bit lower down.)

If you haven’t created the PostgreSQL database yet, follow these steps:

  • Update config/database.yml to look something like this:

  • Start the database server with this command (if you’ve got the default settings)

    $ postgres -D usr/local/var/postgres

  • Create the database

    $ rake db:create

  • Don’t run the migrations. All the necessary statements are in the dumpfile.

At this point you should be able to fill your PostgreSQL database with your SQLite data with this command:

$ psql -d my_database_development -W < db/sqlite-dumpfile201311231200.sql

This invokes the PostgreSQL binary. The -d flag tells it to open a database with the name that follows (in this case “my_database_development”; the name of your database can be found in config/database.yml). Then -W specifies to prompt for your password. This password will be the password for your OSX user account. The last part of the command specifies which file to use to feed in to your new database. This will be the SQLite dumpfile that you created and then modified.

There’s one more step you need to do. If left as is, PostgreSQL will try to add records with the primary key starting at 1, because we removed the sequence information from the dumpfile, as it pertained specifically to SQLite. You won’t be able to add new records to the database because the primary key for new records won’t be unique. It is easy to update the sequence information manually, and you may also be able to include the correct statements into your dumpfile prior to loading it into your PostgreSQL database. Here’s the manual version:

  • Go into the PostgreSQL interactive terminal with your database

    $ psql -d my_database_development

  • I recommend looking at your schema next, so that you know the names of all the tables you have (database=# represents thepsql prompt — the commands to run come after the =#)

    database=# \d

  • You can also check to see where your database will try to start autoincrementing from (but this is not necessary)

    database=# SELECT nextval('ks_users_id_seq');

  • Set the starting point to be one after than the last record in the table. Be sure to use single quotes for the arguments here

    database=# SELECT setval('ks_users_id_seq', (SELECT MAX(id) FROM (ks_users)+1));

  • Repeat this process for every table that has an autoincrementing primary key See this post for a more detailed explanation of this process.

And that’s all. You should be able to access your database with all of your old data and add records with no trouble.

First annual #MagicTheGathering tourney is kicking off w TA @spencer1248&#8217;s beginner info session (at The Flatiron School)

First annual #MagicTheGathering tourney is kicking off w TA @spencer1248’s beginner info session (at The Flatiron School)

Seeds

The following is a guest post by John Richardson and originally appeared on his blog. John is currently in the Ruby-003 class at The Flatiron School. You can follow him on Twitter here.

ADK 46R Logo

Background: ADK 46-R

The ADK 46ers are a set of 46 mountains in upstate NY higher than 4000 feet. If you climb all of these mountains, keep records of your climbs, and then submit those records here, you can become a member of the 46-R club

ADK 46-R CLI

This project started as a Command Line Interface (CLI) that scrapped the following two sites for data on the mountains in the 46-R group and the associated hikes.

http://www.adk46er.org/peaks/index.html

http://www.everytrail.com/guide/adirondack-46er-list

The scrape loaded all the necessary data into a database and the CLI calls from that database when responding to user commands. The CLIT outputs the data in the following format:

forty-sixer-on-rails

The next iteration of this project was using this data as the basis for a rails application that would allow users to view the mountains in the group, view the hikes in which those mountains could be hiked, and then keep track of their progress towards the goal of becoming an ADK 46-R. Before setting up the routes, controllers, views or any of that other good rails stuff, I needed to figure out how to get this data from my CLI applicaiton into my new rails application.

I didn’t want to just add the table to the rails database because I wanted the database to be integrated into ActiveRecord and the models I had designed for the application. I also didn’t want to hand write a massive seed file to get all this data into the database in my rails application.

Export to CSV

What I ended up doing was exporting the table from database that contained the scrapped data as a csv file and importating that csv file into the mountains and hikes tables in the rails database using the following commands:

I did this step for both the moutains and hikes tables which resulted in following two files: moutnains.csv and hikes.csv. I then went to the database in the forty-sixer rails project and did the following to import these two tables:

The .seperator command was an important step in getting the sqlite3 to undestanding the format of the data I was importing. Another important step was not using MS Excel to open and manipulate the csv file before I tried importating the csv file into the database. Initially I had opened the csv file in Excel and added to columns for created_at and updated_at in order for the columns in the csv file to line up with the columns in the rails database. Long story short, this did not work. What I ended up doing was removing the timestamps from the rails database and remigrating before I imported the csv file so that the columns in the rails databse and the CLI database were identical.

seed_dump

The rails application database constantly gets changed during development often getting dropped or reset. I wanted a way to make a seed file out of the data I had just worked so hard to get into my database. Here is where the ruby gem seed_dump saved the day.

Once installed in your gemfile, the following command:

will make seed files out of all the tables in your rails database. In other words, it took 47 rows (MacNaughton mountain is not on the official list, but its over 4000 ft) of this from the table

and turned it into 47 lines of ruby code in my seed file that looked like this

There are more elegent solution to this problem. For example, I could have added the scrape logic from the CLI application into the rails application and done the scrape directly from the rails applciation. On second thought, that would have been a lot easier. That’s what I get for working on this on the LIRR with no interent connection. At least now I have a backup plan next time I lose access to the internet or the data I need is only available through an existing database.

JavaScript vs. Ruby

The following is a guest post by Amanda Himmelstoss and originally appeared on her blog. Amanda is currently in the Ruby-003 class at The Flatiron School. You can follow her on Twitter here.

After beginning JavaScript last Thursday, I’ve been thinking about the differences between this language and Ruby, whom I’ve gotten quite intimate with these last 7 weeks. Coming from Ruby, JavaScript isn’t all that different, yet the syntax is unfamiliar and some logic differs slightly.

typeof vs class

To determine what type of value something is (a number, a fixnum, a string, an Object, an array, etc), in JavaScript you write typeof and then then the object; in Ruby, you append .class to the end of the object.

=== vs ==

In JavaScript, the triple equals sign helps determine if two objects are the same exact object (having the same typeof and the same value). For example: 3 === 3 returns true In Ruby, that is accomplished through the double equals. (3 == 3 returns true)

JavaScript has its own double equals, which shouldn’t be confused with Ruby’s: double equals in JavaScript only determines if the values match. For example: "3" == 3 returns true

parseInt(); vs .to_i

In JavaScript, to turn a string “10” into the integer 10, you write parseInt("10");. In Ruby, you write "10".in_i.

Incrementing and Decrementing

To increment or decrement by 1 in JavaScript, you can write ++ or -- (for example, var x = 2; x++; returns 3. In Ruby, it’s x += 1. You can write this as well in JavaScript (which is useful if you want to increment with numbers other than 1), however, you have the handy ++ and -- as well.

Functions

Functions in JavaScript are the same as methods in Ruby. The syntax of declaring a function vs. declaring a method is slightly different. In Ruby, it’s :

In JavaScript, the same logic can be written as:

Objects

Objects (similar to Ruby’s hash), declared as variables, are a way of organizing key/value pairs. For example, var cat = {color: "grey", name: "Spot", age: "12"}; To access the value of an associated key, you can call that on the variable name: cat.name returns "Spot". You can also easily delete a key/value pair by writing delete cat.name. To declare a new key/value pair, write: cat.gender = "male";. This appends this new data point to the end of the Object.

Arrays

Both Ruby and JavaScript have Arrays, which are more of less logically the same; however, the Object Array functions (JavaScript) and the Class Array methods (Ruby) have different syntax. For example, to find the index of an element in an array, in JavaScript you write arrayName.indexOf("element");. In Ruby, this is accomplished by array_name.index("element")

each

The method each in Ruby is a handy way of iterating through an array of elements in order to change or manipulate them. JavaScript as well as an each function that does the same thing; however, the syntax is slightly different:

And in Ruby, the same is written as:

Falsey Values

One big difference between Ruby and JavaScript is the falsey-ness of certain values. Like in Ruby, null (in Ruby, it’s nil) and false will return false, however, some truthful values in Ruby will return false in JavaScript, like 0, empty strings (""), and undefined.

Object Oriented Programming

JavaScript, like Ruby, can be object-oriented, meaning that objects, their associated methods/functions, can be neatly contained. In JavaScript, a Prototype is much like Ruby’s Class. To create a new Prototype in JavaScript:

The maruCat object is considered a new instance of the Cat prototype. Prototype is essentially the same as Class in Ruby. Note that this in JavaScript is the same concept as self in Ruby.

Hide Your Keys, Hide Your Tokens … Unless Deploying to Heroku

The following is a guest post by Greg Eng and originally appeared on his blog. Greg is currently in the Ruby-003 class at The Flatiron School. You can follow him on Twitter here.

In that case, you have to do a little more than just hide them.

Typically, an API will require a key or authentication token before serving data downstream. It’s logical to avoid committing any of these to a public repo on GitHub, so I often found myself falling into a pattern. I would make hidden files in the assets folder of my project, then .gitignore each one.

It would result in a folder structure that might look like this:

Then in the class that controls all of the API calls, I would do something like this:

I set each relevant piece to a constant to be used in methods throughout the API interactions.

This way feels a little clunky, but it works. As long as you remember to add each hidden file to the .gitignore, it keeps the credentials safe. But as a project grows and it accumulates more authorizations, it becomes a little difficult to manage.

It wasn’t until my group tried to deploy an app to Heroku that we discovered the better way. Heroku makes it simple to deploy a Rails app, but as a result it lacks a lot of the bells and whistles. As a result, it is impossible to log in to the server in a shell to upload each individual hidden file described above.

In order to accomplish the same level of security and function, we can instead use Heroku config variables that will be read at runtime. Heroku has some great documentation on this topic.

It’s very easy to set up from the local shell. Thanks to my teammate David Bella, we were able to figure out what we needed to do immediately.

Just navigate to the root of your project directory and run the heroku config:set command.

For example:

That’s all there is to it! If you want to see your setup at any point, just type heroku config to see an output like this:

This is great for production, but how can we get the same level of organization and clarity for our development environment? My classmate Scott Luptowski told us about the dotenv gem that assists with that exact problem.

Start by including gem 'dotenv-rails' in your Gemfile in the appropriate groups. We only use it in :test and :development. Then, make a .env file at the root of your project directory. In it, include the assignments for each of the keys, tokens or passwords you want to call.

For example:

Rails will be able to call these variables anywhere in the application by using this pattern: ENV['API_KEY']ENV['AUTH_TOKEN']ENV['PASSWORD']. The hard coded keys will stay out of your code and you will have one organized place to manage all the necessary information. Just remember to also add the .env file to your .gitignore. The app should now work seamlessly in both local development and production Heroku environments. A simple solution for an important problem!

Formatting Phone Numbers on an iPhone

The following is a guest post by Dulio Denis and originally appeared on his blog. Dulio is currently in the iOS-000 class at The Flatiron School. You can follow him on Twitter here.

With my Corporate Directory CapStone project I found myself needing to format a phone number in phone fields like cell phone, home phone, office phone, etc.  I was surprised to find this is not as easy on an iPhone as you may think.  Even though clearly Apple has this functionality in Contacts they have no Public API for phone formatting.  You may think what’s the big deal anyhow. Well, in the US you could have a leading 1 or not and you want to be consistent with area code segmentation and then there are international numbers.  I started looking on stackoverflow but thought there ought to be a better way than implementing my own NSFormatter.After looking around I found a good GitHub project by Willy Shark, A.K.A. Serheo Shatunov, called SHSPhoneComponent.  The good news is it also has a CocoaPod.

In this post I’ll outline how to integrate this CocoaPod into your project.

1. First, update your Podfile with the below and do a pod update:

2. Then in your Storyboard or Nib file create a UITextFiled for the phone number to be formatted.  I was using UILabels which do not work as these are UITextField and NSFormatter subclasses for formatting phone numbers. Connect your UITextField(s) to the proper View’s Outlet(s). Since we are using a custom class we need to make the @property of the phone labels look something like:

Then using the Identity Inspector set the custom class of the UITextField to this SHSPhoneTextField.

3. In your View’s include the following libraries:

4. In your View’s ViewDidLoad method

One thing to note is that in line 2 above the addOutputPattern: requires a regular expression in order to have a pattern to match. One site I found which helped me test my regex with my data was a regular expression calculator at rubular.