PGP Key Transition

Keybase | Gist

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA512

PGP Key Transition Statement
José Padilla 
Fri Jul 28 12:54:01 UTC 2017

I have created a new OpenPGP key and will be transitioning away from
my old key. The old key has not been compromised and will continue to
be valid for some time, but I prefer all future correspondence to be
encrypted to the new key, and will be making signatures with the new
key going forward.

I would like this new key to be re-integrated into the web of trust.
This message is signed by both keys to certify the transition. My new
and old keys are signed by each other. If you have signed my old key,
I would appreciate signatures on my new key as well, provided that
your signing policy permits that without re-authenticating me.

The old key, which I am transitioning away from, is:

pub   2048R/9B2987B1 2014-03-04
      Key fingerprint = 6120 BB14 9792 D8E9 A371  B03C AAE3 EF57 9B29 87B1

The new key, to which I am transitioning, is:

pub   4096R/B55434E2 2017-07-28
      Key fingerprint = 58FD 4723 5047 E944 BDE3  4DC7 9A11 1405 B554 34E2

I disown all other and prior keys, so please don't use them.
Specifically, the following keys are not valid for me:

* 0x33CFB6D79478C173
* 0x56921E75F4A66D4C
* 0x7C09FCF380E5AFA3
* 0x55FCA69C27265701
* 0xAAE3EF579B2987B1 (as provided above)

The entire key may be downloaded from: https://keybase.io/jpadilla/pgp_keys.asc?fingerprint=58fd47235047e944bde34dc79a111405b55434e2

To fetch the full new key from a public key server using GnuPG, run:

  gpg --keyserver keys.gnupg.net --recv-key B55434E2

If you already know my old key, you can now verify that the new key is
signed by the old one:

  gpg --check-sigs B55434E2

If you are satisfied that you've got the right key, and the User IDs
match what you expect, I would appreciate it if you would sign my key:

  gpg --sign-key B55434E2

You can upload your signatures to a public keyserver directly:

  gpg --keyserver pgp.mit.edu --send-key B55434E2

Or email [email protected] (possibly encrypted) the output from:

  gpg --armor --export B55434E2

If you'd like any further verification or have any questions about the
transition please contact me directly.

/jpadilla
-----BEGIN PGP SIGNATURE-----

iQEcBAEBCgAGBQJZezRFAAoJEKrj71ebKYexuL0H/RG388KJwp+fWA8hR0jMrSFX
OC4zgsMB6GLsSiZJhivDvYTXJdcFc2x6l9Cgh4YQTZwsDLsK2O7+Ao+AxcUAC5rQ
zb2UIX6GzEly7i7+2HNKpv2AKcpdI5yNwuvjvTJMeivDAnSr9ymSTkbzIMyUuGkh
bEuIodVuuf0jvrgECw8I8DLsyKPt3KkVPYjc1Ji3Q0w0uVtQhndvaD1Etc7wO3Vw
h28Rkw7a6bK8Dt/ZmC+PI50/rKMDWPlSV2+ATCpitKZrDvEyk843d4ue39NaZ6uo
rKLn6Fv0kZLZzX2VnEi64TKUfu7s2LzTOoKLT4km6M2ExcT7rj84nNlPCZrICluJ
AhwEAQEKAAYFAll7NEUACgkQN2XJ6h5H7264vRAAtn7Z3rey+Hsi1g7tBEeYs0wY
pjrn3hPQ5Ej1A+V5kpZuTH0Hh8TaW1LxmClRLE0jmMZV9bhRbCfyG9GMaSFeJp1W
NXZAfXW2vtNurCKgsyBJhtk1MNZg7pbURvcaXifq+gt/Z7UZ6X/YlAfPLsmp22e+
wVA7i0TUlXYcQOtNwhhDD2N/hoG1lMgB2XkC8uzvRIOn06kSneFORzegyeK60rBw
GdbzhuQrkzqZ0inhuEIXzDHFA7yIfOKg3aabx4Mc/K++42iy+bBZzx9UDHGoKay3
RSdnzRNCiQAL2S0Ckk+qo56rHzTqsz0XFtaLdMjp/c58knrLnVekMnPMJMXnnxwj
dAzV31Lyxk4VZomw2z2PA/1R/C9bY2msJ5hCggqICguwM2bnRzn17mEawwAKTY0S
rCK/744woagAa57muhVZyQk7eVXKxiS7EVjNGVM83jk0hwuUBv+v55Zawi3hm8fL
RUMXwxIkv+HHD76x//mBmLOrjkdHMlZ1WyPiOFbRQrthpUabZ3V27h7IObsOZcbv
S9e5hh69jEcOZffL/I5iuu4zc6Wcq3F7G5oiMoVGPUSUQUxhZtZX9OK1PzrMQezw
2ejhY3+1ktS5rsPTMw6qjiei0E0MIa+Yluw2u6CeT7oxQlJXuSFKCQB1CAnEOE/l
iWstOpXJiOkvoLtMrPM=
=hyse
-----END PGP SIGNATURE-----

Update Kubernetes Deployment after pushing image to Docker Hub

image

I recently moved FilePreviews.io’s workers deployment to Kubernetes in Google Container Engine. After setting the workers up as a Deployment I wondered how this would fit with my current setup for continuous deployment. On a previous setup, I relied on Docker Hub to automatically build, push tagged images, and notify Rancher via a webhook.

I ended up writing a small server to handle the webhook event from Docker Hub and PATCH the container’s image tags in the Kubernetes deployment which triggers a rollout.

Developer Ergonomics

These are the slides for my PyCaribbean 2017 keynote on Developer Ergonomics where I talk about the current state of package managers in Python vs in other ecosystems like Node and Rust.

Looking backward and forward

2016 was with no doubt an eventful year for many.

On March 12th, I married my high school sweetheart. We’d been together for around 9 years and engaged for like 4 of those. Best day ever.

During the last quarter of 2015, we cofounded Alias Payments to build Gasolina Móvil. We became part of Parallel18′s first cohort and after six months we were closing a deal with PumaEnergy.

On August, my wife and I packed our things and moved to Hartford, CT for her doctoral internship. Its our first time living away from Puerto Rico, but we’re making it work. I’m still finding the right balance when working mostly from home. Since living in the East Coast, we drove to Québec City for my wife’s birthday, visited beautiful Boston way too many times, spent Thanksgiving in NYC with new friends, visited Stars Hollow(Washington Depot, CT), hiked some local trails, visited the casino at Mohegan Sun and more.

During 2016, Blimp was rebranded as a consulting company with products of it’s own. We knew we could help more companies and digital agencies build awesome things and we did. We made our first two hires, a developer and a project manager. We’ve worked on many interesting consulting projects during this year and the next couple of months look very promising.

Moving forward

During the past couple of months I’ve had to split my time between consulting projects and our ongoing products. I let this constant context switching burn me out constantly for a while there. As a result of this, I’ll start dedicating most of my time to our products and phase out from the consulting part of the business.

There’s no doubt I love building products and somewhere in there is where I’m best at and have the most impact. This is why it makes the most sense to me to spend more time doing so.

Last year I also contributed way less to open source projects, mostly because of the lack of time. Looking forward to have some more time again to do so.

On February, I’ll be speaking at PyCaribbean. If you haven’t get your tickets now, this year it’ll be happening in Puerto Rico!

On March, I’ll be attending EmberConf with @gcollazo.

Sometime later in 2017, my wife is graduating. Hello future Dr. Conde-Padilla PsyD. I’m very proud of you!

Here’s to family and friends, being healthy, happy, and better, more learning, more reading. Here’s to 2017.

Upload files to S3 and generate previews using Laravel

I recently put together a PHP client library for FilePreviews and immediately thought about putting together a blog post on how I’d use it. After 6 years, according to this repo, of not writing a single line of PHP, I looked into Laravel since it seems to be the rave these days. Alright, let’s get to it.

This is a step by step guide on how to use Laravel to upload files to S3, and generate previews and extract metadata using FilePreviews.io. If you are already uploading files to S3 with Laravel, check out how to integrate with FilePreviews.

Create a Laravel project

I’m assuming you’ll probably have composer already installed.

$ composer create-project laravel/laravel --prefer-dist filepreviews-laravel-example

Filesystem / Cloud Storage Setup

In this example we’ll be using AWS S3 to store our files. After you’ve got a bucket and some credentials, let’s setup the project to use them.

First we’ll add S3 support to Laravel

$ composer require league/flysystem-aws-s3-v3

Since no one want’s to commit their S3 credentials we’ll modify config/filesystems.php to use environment variables.

's3' => [
    'driver' => 's3',
    'key'    => env('S3_KEY'),
    'secret' => env('S3_SECRET'),
    'region' => env('S3_REGION'),
    'bucket' => env('S3_BUCKET'),
],

Now add your corresponding credentials to your project’s .env file.

S3_KEY=YOUR_AWS_S3_ACCESS_KEY
S3_SECRET=YOUR_AWS_S3_SECRET_KEY
S3_REGION=us-east-1
S3_BUCKET=YOUR_AWS_S3_BUCKET

Database

We’ll use sqlite in this example instead of Laravel’s default which is MySQL. Set DB_CONNECTION to sqlite in your project’s .env file.

You’ll also need to create an empty sqlite database file: storage/database.sqlite.

Model

Now we need to create a Document model. Our Document model will have a name, file(a URL on S3 to our original file), preview_url(a URL to a preview generated by FilePreviews), and preview(a JSON string containing all the results and metadata generated by FilePreviews.

First we’ll generate our model and it’s pertaining migration file.

$ php artisan make:model Document --migration

We need to add the rest of our fields to that migration before running it. You’ll find the migration inside the database/migrations directory. It’ll be named something like 2015_11_25_125309_create_documents_table.php.

Make sure the up() function looks like the following.

public function up()
{
    Schema::create('documents', function (Blueprint $table) {
        $table->increments('id');
        $table->string('name');
        $table->string('file');
        $table->string('preview_url')->nullable();
        $table->json('preview')->nullable();
        $table->timestamps();
    });
}

Now run your migrations.

$ php artisan migrate

Controllers

Our simple application will allow you to list existing documents and create new ones. To simplify things a bit we’ll generate a resource controller.

$ php artisan make:controller DocumentController

Let’s go ahead and add that to our routes.

Route::resource('documents', 'DocumentController');

Next we’ll configure our controller’s functions, index which will render our list of documents, and create which will render our form.

<?php

namespace AppHttpControllers;

use IlluminateHttpRequest;

use AppDocument;
use AppHttpRequests;
use AppHttpControllersController;

class DocumentController extends Controller
{
    /**
     * Display a listing of the resource.
     *
     * @return IlluminateHttpResponse
     */
    public function index()
    {
        $documents = Document::all();

        return view('documents.index', compact('documents'));
    }

    /**
     * Show the form for creating a new resource.
     *
     * @return IlluminateHttpResponse
     */
    public function create()
    {
        return view('documents.create');
    }
}

Note: Make sure to import/alias our Document model by adding use AppDocument;. Things like this always got me.

Views

Now our controller is missing the templates/views for both index and create.

Create resources/views/documents/index.blade.php. All this view does is list created documents, showing the id, name, and preview_url if available. We’ll also add a link to our create route.

<html>
<head></head>
<body>
  <h1>Documents</h1>
  <p><a href="{{ route('documents.create') }}">Create Document</a></p>
  @foreach ($documents as $document)
    <ul id="document-{{ $document->id }}">
      <li>ID: {{ $document->id }}</li>
      <li>Name: <a href="{{ $document->url }}">{{ $document->name }}</a></li>
      <li class="preview-url">
        @if ($document->preview_url)
            <a href="{{ $document->preview_url }}">Preview</a>
        @else
          No Preview
        @endif
      </li>
    </ul>
  @endforeach
</body>
</html>

Create resources/views/documents/create.blade.php. All this view does is show any errors with the form, and allow submitting a form with a file input field.

<html>
<head></head>
<body>
  <h1>Create Document</h1>
  @if (count($errors) > 0)
    <div>
      <ul>
        @foreach ($errors->all() as $error)
          <li>{{ $error }}</li>
        @endforeach
      </ul>
    </div>
  @endif

  <form action="/documents" method="POST" enctype="multipart/form-data">
    {{ csrf_field() }}

    <p>
      <label for="file">File</label>
      <input type="file" name="file">
    </p>

    <input type="submit">
  </form>
</body>
</html>

Note: Make sure the form has the correct enctype or else you’ll have issues uploading files.

Uploading to S3 and storing Document

Now the fun starts! Let’s add the store function to our DocumentController.

<?php

namespace AppHttpControllers;

use Storage;
use IlluminateHttpRequest;

use AppDocument;
use AppHttpRequests;
use AppHttpControllersController;

class DocumentController extends Controller
{
    /**
     * Display a listing of the resource.
     *
     * @return IlluminateHttpResponse
     */
    public function index()
    {
        $documents = Document::all();

        return view('documents.index', compact('documents'));
    }

    /**
     * Show the form for creating a new resource.
     *
     * @return IlluminateHttpResponse
     */
    public function create()
    {
        return view('documents.create');
    }

    /**
     * Store a newly created resource in storage.
     *
     * @param  IlluminateHttpRequest  $request
     * @return IlluminateHttpResponse
     */
    public function store(Request $request)
    {
        $this->validate($request, [
            'file' => 'required'
        ]);

        $file = $request->file('file');

        if ($file->isValid()) {
            $name = $file->getClientOriginalName();
            $key = 'documents/' . $name;
            Storage::disk('s3')->put($key, file_get_contents($file));

            $document = new Document;

            $document->name = $name;
            $document->file = $key;
            $document->save();
        }

        return redirect('documents');
    }
}

Note: Make sure to import/alias our Document model by adding use Storage;.

A couple of things going on here, let’s break it down.

  1. Validate our file field is present.
  2. Make sure file is valid.
  3. Store file in S3. This file stored on S3 are private by default, meaning that we’ll need to generate a temporary URL to access them.
  4. Create Document using file name and key where file was stored in S3.
  5. Redirect to /documents.

Since our files are stored privately on S3 we’ll need a way to generate that temporary secure URL to access them. A good place to add this is in our model. We’ll append a dynamic property to our model called url. By using the underlying AWS client library, we can create a presigned URL that expires in 20 minutes. We access this property like any other, e.g. $document->url.

<?php

namespace App;

use Storage;
use Config;
use IlluminateDatabaseEloquentModel;

class Document extends Model
{
    protected $appends = ['url'];

    public function getUrlAttribute()
    {
        return $this->getFileUrl($this->attributes['file']);
    }

    private function getFileUrl($key) {
        $s3 = Storage::disk('s3');
        $client = $s3->getDriver()->getAdapter()->getClient();
        $bucket = Config::get('filesystems.disks.s3.bucket');

        $command = $client->getCommand('GetObject', [
            'Bucket' => $bucket,
            'Key' => $key
        ]);

        $request = $client->createPresignedRequest($command, '+20 minutes');

        return (string) $request->getUri();
    }
}

If you run php artisan serve and navigate to http://localhost:8000/documents/create, pick a file, and submit, you should be redirected to http://localhost:8000/documents. You’ll now see the created Document. If you click on the file’s name you should be redirected to your file on S3 using our presigned URL. Success!

So we still haven’t actually done anything with FilePreviews at this point. This example until nows serves as an idea of what most people would be actually doing already. Now we’ll see how FilePreviews fits into all this.

Integrating with FilePreviews.io

We’ve put together a package that’ll help you implement FilePreviews in your Laravel projects. Let’s install that.

$ composer require filepreviews/filepreviews-laravel

Now we need to add our Service Provider and Facade to config/app.php.

'providers' => [
    // ...

    FilePreviewsLaravelFilePreviewsServiceProvider::class,
],

'aliases' => [
    // ...

    'FilePreviews' => FilePreviewsLaravelFilePreviewsFacade::class,
]

To customize the configuration file, publish the package configuration by running php artisan vendor:publish.

Signup for a FilePreviews.io account by going to https://api.filepreviews.io/auth/signup/. Create a free application and get your Server API Key and Server API Secret from you application’s settings. Add those to your project’s .env.

FILEPREVIEWS_API_KEY=YOUR_FILEPREVIEWS_SERVER_API_KEY
FILEPREVIEWS_API_SECRET=YOUR_FILEPREVIEWS_SERVER_SECRET_KEY

Generating Preview

Now we need to request generating a preview after our file has been uploaded and our Document created. Let’s add a function to our model which will then call from our controller.

public function requestPreview()
{
    $fp = app('FilePreviews');

    $options = [
        'metadata' => ['checksum', 'ocr'],
        'data' => [
            'document_id' => $this->attributes['id']
        ]
    ];

    $url = $this->getFileUrl($this->attributes['file']);

    return $fp->generate($url, $options);
}

We’re asking FilePreviews to generate a request for a file, extract the file’s checksum and OCR. We’re also adding our document id. We’ll use this later so we can identify what preview/metadata belongs to what document.

Let’s tweak our controller’s store function.

public function store(Request $request)
{
    $this->validate($request, [
        'file' => 'required'
    ]);

    $file = $request->file('file');

    if ($file->isValid()) {
        $name = $file->getClientOriginalName();
        $key = 'documents/' . $name;
        Storage::disk('s3')->put($key, file_get_contents($file));

        $document = new Document;

        $document->name = $name;
        $document->file = $key;
        $document->save();

        $document->requestPreview();
    }

    return redirect('documents');
}

After creating our document, we are calling our $document->requestPreview() to let FilePreviews know what we want.

If you run php artisan serve and navigate to http://localhost:8000/documents/create, pick a file, and submit, you should be redirected to http://localhost:8000/documents. You’ll now see the created Document, but you’ll still see “No Preview”. Why is that?

If you recall our requestPreview() function we call $fp->generate($url, $options);. This lets FilePreviews know what we want extracted from our file, we get a confirmation, but no results just yet. This is a fire and forget operation, since it could possibly take a few minutes depending on the file size and requested metadata. We could poll FilePreviews for our results but that’s not really efficient. This is why FilePreviews allows you to subscribe to your application’s webhook by setting a Callback URL.

Webhooks

Let’s go ahead and setup our application to handle webhooks. With filepreviews-laravel this is really easy. Add the following route to app/Http/routes.php:

Route::post('filepreviews/webhook', '[email protected]');

Since FilePreviews webhooks need to bypass Laravel’s CSRF verification, be sure to list the URI as an exception in your app/Http/MiddlewareVerifyCsrfToken.php middleware:

<?php

namespace AppHttpMiddleware;

use IlluminateFoundationHttpMiddlewareVerifyCsrfToken as BaseVerifier;

class VerifyCsrfToken extends BaseVerifier
{
    /**
     * The URIs that should be excluded from CSRF verification.
     *
     * @var array
     */
    protected $except = [
        'filepreviews/webhook'
    ];
}

The FilePreviews webhook controller we registered fires two events: filepreviews.success and filepreviews.error. Let’s create a Listener(app/Listeners/FilePreviewsSuccess.php) to handle the filepreviews.success event.

<?php

namespace AppListeners;

use Event;

use AppDocument;

class FilePreviewsSuccess
{
    /**
     * Handle the event.
     *
     * @param  array  $results
     * @return void
     */
    public function handle($results)
    {
        $document_id = $results['user_data']['document_id'];
        $document = Document::find($document_id);
        $document->preview_url = $results['preview']['url'];
        $document->preview = json_encode($results);
        $document->save();
    }
}

Let’s setup the event listener mappings for this application in app/Providers/EventServiceProvider.php.


protected $listen = [
        'filepreviews.success' => [
            'AppListenersFilePreviewsSuccess'
    ]
];

Our listener will react to the filepreviews.success event, look for a Document that matches document_id and update it.

To try this out locally I recommend using something like ngrok.

With php artisan serve running, on another tab/window run ngrok localhost:8000. Setup ngrok’s forwarding URL as your Callback URL on your FilePreviews application’s settings. Once that’s done navigate to http://localhost:8000/documents/create, pick a file, and submit. In a few seconds you should see a request logged in ngrok with the status 200 OK. If you then navigate to http://localhost:8000/documents you should see a link “Preview” instead of “No Preview”. Click that and you’ll see an image of the file you previously uploaded. Super cool stuff, right?

Bonus: Realtime with Pusher

Just because I’m really liking Laravel, I’ll show you how to add some realtime goodness using Pusher to what we already have working. Once we’re done we won’t need to refresh to see whenever our documents get previews.

Let’s add support for broadcasting events to Pusher.

$ composer require pusher/pusher-php-server

Go signup for a free account on Pusher, create an app, and set credentials in your project’s .env.

PUSHER_KEY=YOUR_PUSHER_APP_KEY
PUSHER_SECRET=YOUR_PUSHER_APP_SECRET
PUSHER_APP_ID=YOUR_PUSHER_APP_ID

Let’s create an event app/Events/FilePreviewsGenerated.php. This event will broadcast on the filepreviews channel an event called filepreviews.generated

<?php

namespace AppEvents;

use IlluminateQueueSerializesModels;
use IlluminateContractsBroadcastingShouldBroadcast;

use AppEventsEvent;
use AppDocument;

class FilePreviewsGenerated extends Event implements ShouldBroadcast
{
    use SerializesModels;

    public $document;

    /**
     * Create a new event instance.
     *
     * @return void
     */
    public function __construct(Document $document)
    {
        $this->document = $document;
    }

    /**
     * Get the channels the event should be broadcast on.
     *
     * @return array
     */
    public function broadcastOn()
    {
        return ['filepreviews'];
    }

    /**
     * Get the broadcast event name.
     *
     * @return string
     */
    public function broadcastAs()
    {
        return 'filepreviews.generated';
    }
}

We’ll tweak our FilePreviewsSuccess.php listener to fire a FilePreviewsGenerated event after saving our document.

<?php

namespace AppListeners;

use Event;

use AppDocument;
use AppEventsFilePreviewsGenerated;

class FilePreviewsSuccess
{
    /**
     * Handle the event.
     *
     * @param  array  $results
     * @return void
     */
    public function handle($results)
    {
        $document_id = $results['user_data']['document_id'];
        $document = Document::find($document_id);
        $document->preview_url = $results['preview']['url'];
        $document->preview = json_encode($results);
        $document->save();

        Event::fire(new FilePreviewsGenerated($document));
    }
}

All that’s left is add the Pusher JavaScript code on our index view.

<html>
<head></head>
<body>
  <h1>Documents</h1>
  <p><a href="{{ route('documents.create') }}">Create Document</a></p>
  @foreach ($documents as $document)
    <ul id="document-{{ $document->id }}">
      <li>ID: {{ $document->id }}</li>
      <li>Name: <a href="{{ $document->url }}">{{ $document->name }}</a></li>
      <li class="preview-url">
        @if ($document->preview_url)
            <a href="{{ $document->preview_url }}">Preview</a>
        @else
          No Preview
        @endif
      </li>
    </ul>
  @endforeach

  <script src="https://code.jquery.com/jquery-1.11.3.min.js"></script>
  <script src="https://js.pusher.com/3.0/pusher.min.js"></script>
  <script>
    $(function() {
      // Enable pusher logging - don't include this in production
      Pusher.log = function(message) {
        if (window.console && window.console.log) {
          window.console.log(message);
        }
      };

      var pusherKey = '{{ config('broadcasting.connections.pusher.key') }}',
          pusher = new Pusher(pusherKey, { encrypted: true }),
          channel = pusher.subscribe('filepreviews');

      channel.bind('filepreviews.generated', function(data) {
        var $doc = $('#document-' + data.document.id),
            $previewUrl = $doc.find('.preview-url');

        $previewUrl.html('<a href="' + data.document.preview_url + '">Preview</a>');
      });
    });
  </script>
</body>
</html>

With php artisan serve and ngrok running, we’ll need a new tab/window to run php artisan queue:listen.

The complete source code for this example project is available on GitHub.

If you have any questions or feedback on this post or FilePreviews.io, feel free to let me know.