Using Replicate Gem With Heroku

Heroku vs Replicate

Update 01/2013:

I’ve recently found that the heroku run command can randomly truncate output so your milage on this may vary, especially with larger sets of data. GitHub Issue here: https://github.com/heroku/heroku/issues/674

While trying to use the Replicate gem with Heroku, I ran into a strange issue. Namely, Heroku didn’t like the bytestream that Ruby’s Marshal class was producing:

1
2
3
4
5
6

╰─○  heroku run 'replicate -r ./config/environment -d "Post.first"' > file.dump
 !    Heroku client internal error.
 !    Search for help at: https://help.heroku.com
 !    Or report a bug at: https://github.com/heroku/heroku/issues/new

    Error:       invalid byte sequence in UTF-8 (ArgumentError)

Workaround

I did find a workaround though. The base64 binary is present on Heroku VMs so you can pipe replicate’s output through that to get data that Heroku can handle. However, there are a couple caveats here:

  1. Heroku squashes STDERR into STDOUT which means you need to filter out STDERR (via 2>/dev/null) before sending the data back to the client.

  2. You also need to do a little manual work on the data you get back to filter out the lines Heroku addes to the output.

  3. Decode the base64 file on your end before passing it your local replicate instance.

Example

Here is the flow that works for me:

1

heroku run 'replicate -r ./config/environment -d "Post.first" 2>/dev/null | base64' > file.dump

Open file.dump and remove any of the “…attached to terminal” lines Heroku adds. In the following case, just the first line.

1
2
3

Running `replicate -r ./config/environment -d "Post.first" 2>/dev/null | base64` attached to terminal... up, run.7145
BAhbCEkiCVVzZXIGOgZFRmkGewlJIgdpZAY7AFRpBkkiDXVzZXJuYW1lBjsAVEkiCEJvYgY7AFRJ
Ig9jcmVhdGVkX2F0BjsAVFU6IEFjdGl2ZVN1cHBvcnQ6OlRpbWVXaXRoWm9uZVsISXU6CVRpbWUN

Decode the base64 file back into a Marshal friendly file and load it into Replicate:

1
2
3
4
5

╰─○ base64 -i file.dump -D | replicate -r ./config/environment -l
zsh: correct './config/environment' to './config/environments' [nyae]? n
==> loaded 2 total objects:
Post      1
User      1

Hope that helps anyone working with Replicate on Heroku.

Rails UJS and jQuery Mobile

I ran into a gotcha when working with jQuery Mobile and Rails UJS helpers. As you probably know, adding remote: true to your form_for arguments triggers Rails to add some UJS helpers that submit that form via AJAX rather than a normal post.

However, jQuery Mobile looks for any form element and does the same. As a result, I was getting double posts each time I clicked submit on what should have been a very simple form.

Adding a data attribute of data-ajax='false' prevents jQuery Mobile from trying to muck with the form at all and lets Rails do its thing.

OpenStack Simple Scheduler

Interesting gotcha found while playing with OpenStack.

We were working on a two node OpenStack cluster but ran into a problem where the scheduler was not properly balancing between the two nodes. We’d spin up 10 instances and eight would spawn on one box and two on the other. Not ideal, obviously.

We figured out that you can specify a different scheduler via nova.conf. We added:

1

--scheduler_driver=nova.scheduler.simple.SimpleScheduler

Despite its name, SimpleScheduler tries to intelligently schedule new instances based on the current load of the available compute nodes. That solved our issue!

This isn’t very well documented. I had to go digging through the OpenStack code to figure out the syntax we needed but that’s why open source is awesome. :-)

Overall, OpenStack has blown me away. Awesome stuff.

Red Hat Needs a Build-essentials Package

Dear Red Hat,

Please fix this:

1
2
3
4
5
6
7
8

[root@localhost ~]# yum groupinstall "Development Tools"
....
Transaction Summary
=================
Install 133 Package(s)
Upgrade 9 Package(s)

Total download size: 137 M

vs Ubuntu

1
2
3
4
5
6
7
8
9

user@ubuntu:~$ sudo apt-get install build-essential
Reading package lists... Done
Building dependency tree 
Reading state information... Done
....
0 upgraded, 16 newly installed, 0 to remove and 0 not upgraded.
Need to get 19.6MB of archives.
After this operation, 66.8MB of additional disk space will be used.

19.6MB to your 137MB.

I just want the packages I need to compile and install software, I don’t want every development library available on the system.

Your Example Code Doesn’t Instill Confidence…

1
2
3
4
5
6

sendClass = SendClassification.new(nil,nil,nil,nil,nil,nil,nil,189,nil,nil,nil,nil,nil,nil,nil,nil)
sendProf = SenderProfile.new(nil,nil,nil,nil,nil,nil,nil,352,nil,nil,nil,nil,'NTO Customer Support','acruz@example.com',nil,nil,nil,nil,nil,nil,nil,nil,nil,nil,nil,nil)
delProf = DeliveryProfile.new(nil,nil,nil,nil,nil,nil,nil,306,nil,nil,nil,nil,nil,nil,nil,nil,nil,nil,nil,nil,nil,nil)
email = Email.new(nil,nil,nil,nil,nil,2052608,nil,nil,nil,nil,nil,nil,nil,nil,nil,nil,nil,nil,nil,nil,nil,nil,nil,nil,nil)
sendDefList = SendDefinitionList.new(nil,nil,nil,nil,nil,nil,nil,nil,nil,nil,nil,nil,objID,'CustomObject',nil,nil,nil,nil,nil)
emailSendDef = EmailSendDefinition.new(nil,nil,nil,nil,nil,nil,nil,customerKey,nil,nil,customerKey,desc,nil,nil,nil,sendClass,sendProf,nil,nil,delProf,nil,nil,nil,nil,nil,nil,nil,nil,0,0,sendDefList,email,nil,nil,nil,'Thank You for Contacting NTO Customer Support',nil,1,1,nil,nil,nil,nil,1,nil,nil,nil,'acruz@example.com'

User Interface Magic

In Lotus Notes, this is what one is prompted with after double clicking an attachment in an email.

There are really 2 things users want to do with email attachments:

  1. Open it
  2. Download it

They’ve managed to split two actions into five buttons. Huzzah! It’s bad UI magic!

Apprently, the edit button is in case you want to edit something and then have Notes automatically write the changes back to the email. What?? Why would anyone use their email as storage for a document?

This is one of Notes’ (many) problems. It tries to be everything to everyone and succeeds at nothing in the process.

One simple feature is better than 10 complicated ones. For example, Gmail doesn’t even support anything other than reverse chronological order in the list view. Does anyone really care? No. Because it’s fast and simple.