Upload files to S3 and generate previews using Laravel

I recently put together a PHP client library for FilePreviews and immediately thought about putting together a blog post on how I’d use it. After 6 years, according to this repo, of not writing a single line of PHP, I looked into Laravel since it seems to be the rave these days. Alright, let’s get to it.

This is a step by step guide on how to use Laravel to upload files to S3, and generate previews and extract metadata using FilePreviews.io. If you are already uploading files to S3 with Laravel, check out how to integrate with FilePreviews.

Create a Laravel project

I’m assuming you’ll probably have composer already installed.

$ composer create-project laravel/laravel --prefer-dist filepreviews-laravel-example

Filesystem / Cloud Storage Setup

In this example we’ll be using AWS S3 to store our files. After you’ve got a bucket and some credentials, let’s setup the project to use them.

First we’ll add S3 support to Laravel

$ composer require league/flysystem-aws-s3-v3

Since no one want’s to commit their S3 credentials we’ll modify config/filesystems.php to use environment variables.

's3' => [
    'driver' => 's3',
    'key'    => env('S3_KEY'),
    'secret' => env('S3_SECRET'),
    'region' => env('S3_REGION'),
    'bucket' => env('S3_BUCKET'),
],

Now add your corresponding credentials to your project’s .env file.

S3_KEY=YOUR_AWS_S3_ACCESS_KEY
S3_SECRET=YOUR_AWS_S3_SECRET_KEY
S3_REGION=us-east-1
S3_BUCKET=YOUR_AWS_S3_BUCKET

Database

We’ll use sqlite in this example instead of Laravel’s default which is MySQL. Set DB_CONNECTION to sqlite in your project’s .env file.

You’ll also need to create an empty sqlite database file: storage/database.sqlite.

Model

Now we need to create a Document model. Our Document model will have a name, file(a URL on S3 to our original file), preview_url(a URL to a preview generated by FilePreviews), and preview(a JSON string containing all the results and metadata generated by FilePreviews.

First we’ll generate our model and it’s pertaining migration file.

$ php artisan make:model Document --migration

We need to add the rest of our fields to that migration before running it. You’ll find the migration inside the database/migrations directory. It’ll be named something like 2015_11_25_125309_create_documents_table.php.

Make sure the up() function looks like the following.

public function up()
{
    Schema::create('documents', function (Blueprint $table) {
        $table->increments('id');
        $table->string('name');
        $table->string('file');
        $table->string('preview_url')->nullable();
        $table->json('preview')->nullable();
        $table->timestamps();
    });
}

Now run your migrations.

$ php artisan migrate

Controllers

Our simple application will allow you to list existing documents and create new ones. To simplify things a bit we’ll generate a resource controller.

$ php artisan make:controller DocumentController

Let’s go ahead and add that to our routes.

Route::resource('documents', 'DocumentController');

Next we’ll configure our controller’s functions, index which will render our list of documents, and create which will render our form.

<?php

namespace AppHttpControllers;

use IlluminateHttpRequest;

use AppDocument;
use AppHttpRequests;
use AppHttpControllersController;

class DocumentController extends Controller
{
    /**
     * Display a listing of the resource.
     *
     * @return IlluminateHttpResponse
     */
    public function index()
    {
        $documents = Document::all();

        return view('documents.index', compact('documents'));
    }

    /**
     * Show the form for creating a new resource.
     *
     * @return IlluminateHttpResponse
     */
    public function create()
    {
        return view('documents.create');
    }
}

Note: Make sure to import/alias our Document model by adding use AppDocument;. Things like this always got me.

Views

Now our controller is missing the templates/views for both index and create.

Create resources/views/documents/index.blade.php. All this view does is list created documents, showing the id, name, and preview_url if available. We’ll also add a link to our create route.

<html>
<head></head>
<body>
  <h1>Documents</h1>
  <p><a href="{{ route('documents.create') }}">Create Document</a></p>
  @foreach ($documents as $document)
    <ul id="document-{{ $document->id }}">
      <li>ID: {{ $document->id }}</li>
      <li>Name: <a href="{{ $document->url }}">{{ $document->name }}</a></li>
      <li class="preview-url">
        @if ($document->preview_url)
            <a href="{{ $document->preview_url }}">Preview</a>
        @else
          No Preview
        @endif
      </li>
    </ul>
  @endforeach
</body>
</html>

Create resources/views/documents/create.blade.php. All this view does is show any errors with the form, and allow submitting a form with a file input field.

<html>
<head></head>
<body>
  <h1>Create Document</h1>
  @if (count($errors) > 0)
    <div>
      <ul>
        @foreach ($errors->all() as $error)
          <li>{{ $error }}</li>
        @endforeach
      </ul>
    </div>
  @endif

  <form action="/documents" method="POST" enctype="multipart/form-data">
    {{ csrf_field() }}

    <p>
      <label for="file">File</label>
      <input type="file" name="file">
    </p>

    <input type="submit">
  </form>
</body>
</html>

Note: Make sure the form has the correct enctype or else you’ll have issues uploading files.

Uploading to S3 and storing Document

Now the fun starts! Let’s add the store function to our DocumentController.

<?php

namespace AppHttpControllers;

use Storage;
use IlluminateHttpRequest;

use AppDocument;
use AppHttpRequests;
use AppHttpControllersController;

class DocumentController extends Controller
{
    /**
     * Display a listing of the resource.
     *
     * @return IlluminateHttpResponse
     */
    public function index()
    {
        $documents = Document::all();

        return view('documents.index', compact('documents'));
    }

    /**
     * Show the form for creating a new resource.
     *
     * @return IlluminateHttpResponse
     */
    public function create()
    {
        return view('documents.create');
    }

    /**
     * Store a newly created resource in storage.
     *
     * @param  IlluminateHttpRequest  $request
     * @return IlluminateHttpResponse
     */
    public function store(Request $request)
    {
        $this->validate($request, [
            'file' => 'required'
        ]);

        $file = $request->file('file');

        if ($file->isValid()) {
            $name = $file->getClientOriginalName();
            $key = 'documents/' . $name;
            Storage::disk('s3')->put($key, file_get_contents($file));

            $document = new Document;

            $document->name = $name;
            $document->file = $key;
            $document->save();
        }

        return redirect('documents');
    }
}

Note: Make sure to import/alias our Document model by adding use Storage;.

A couple of things going on here, let’s break it down.

  1. Validate our file field is present.
  2. Make sure file is valid.
  3. Store file in S3. This file stored on S3 are private by default, meaning that we’ll need to generate a temporary URL to access them.
  4. Create Document using file name and key where file was stored in S3.
  5. Redirect to /documents.

Since our files are stored privately on S3 we’ll need a way to generate that temporary secure URL to access them. A good place to add this is in our model. We’ll append a dynamic property to our model called url. By using the underlying AWS client library, we can create a presigned URL that expires in 20 minutes. We access this property like any other, e.g. $document->url.

<?php

namespace App;

use Storage;
use Config;
use IlluminateDatabaseEloquentModel;

class Document extends Model
{
    protected $appends = ['url'];

    public function getUrlAttribute()
    {
        return $this->getFileUrl($this->attributes['file']);
    }

    private function getFileUrl($key) {
        $s3 = Storage::disk('s3');
        $client = $s3->getDriver()->getAdapter()->getClient();
        $bucket = Config::get('filesystems.disks.s3.bucket');

        $command = $client->getCommand('GetObject', [
            'Bucket' => $bucket,
            'Key' => $key
        ]);

        $request = $client->createPresignedRequest($command, '+20 minutes');

        return (string) $request->getUri();
    }
}

If you run php artisan serve and navigate to http://localhost:8000/documents/create, pick a file, and submit, you should be redirected to http://localhost:8000/documents. You’ll now see the created Document. If you click on the file’s name you should be redirected to your file on S3 using our presigned URL. Success!

So we still haven’t actually done anything with FilePreviews at this point. This example until nows serves as an idea of what most people would be actually doing already. Now we’ll see how FilePreviews fits into all this.

Integrating with FilePreviews.io

We’ve put together a package that’ll help you implement FilePreviews in your Laravel projects. Let’s install that.

$ composer require filepreviews/filepreviews-laravel

Now we need to add our Service Provider and Facade to config/app.php.

'providers' => [
    // ...

    FilePreviewsLaravelFilePreviewsServiceProvider::class,
],

'aliases' => [
    // ...

    'FilePreviews' => FilePreviewsLaravelFilePreviewsFacade::class,
]

To customize the configuration file, publish the package configuration by running php artisan vendor:publish.

Signup for a FilePreviews.io account by going to https://api.filepreviews.io/auth/signup/. Create a free application and get your Server API Key and Server API Secret from you application’s settings. Add those to your project’s .env.

FILEPREVIEWS_API_KEY=YOUR_FILEPREVIEWS_SERVER_API_KEY
FILEPREVIEWS_API_SECRET=YOUR_FILEPREVIEWS_SERVER_SECRET_KEY

Generating Preview

Now we need to request generating a preview after our file has been uploaded and our Document created. Let’s add a function to our model which will then call from our controller.

public function requestPreview()
{
    $fp = app('FilePreviews');

    $options = [
        'metadata' => ['checksum', 'ocr'],
        'data' => [
            'document_id' => $this->attributes['id']
        ]
    ];

    $url = $this->getFileUrl($this->attributes['file']);

    return $fp->generate($url, $options);
}

We’re asking FilePreviews to generate a request for a file, extract the file’s checksum and OCR. We’re also adding our document id. We’ll use this later so we can identify what preview/metadata belongs to what document.

Let’s tweak our controller’s store function.

public function store(Request $request)
{
    $this->validate($request, [
        'file' => 'required'
    ]);

    $file = $request->file('file');

    if ($file->isValid()) {
        $name = $file->getClientOriginalName();
        $key = 'documents/' . $name;
        Storage::disk('s3')->put($key, file_get_contents($file));

        $document = new Document;

        $document->name = $name;
        $document->file = $key;
        $document->save();

        $document->requestPreview();
    }

    return redirect('documents');
}

After creating our document, we are calling our $document->requestPreview() to let FilePreviews know what we want.

If you run php artisan serve and navigate to http://localhost:8000/documents/create, pick a file, and submit, you should be redirected to http://localhost:8000/documents. You’ll now see the created Document, but you’ll still see “No Preview”. Why is that?

If you recall our requestPreview() function we call $fp->generate($url, $options);. This lets FilePreviews know what we want extracted from our file, we get a confirmation, but no results just yet. This is a fire and forget operation, since it could possibly take a few minutes depending on the file size and requested metadata. We could poll FilePreviews for our results but that’s not really efficient. This is why FilePreviews allows you to subscribe to your application’s webhook by setting a Callback URL.

Webhooks

Let’s go ahead and setup our application to handle webhooks. With filepreviews-laravel this is really easy. Add the following route to app/Http/routes.php:

Route::post('filepreviews/webhook', '[email protected]');

Since FilePreviews webhooks need to bypass Laravel’s CSRF verification, be sure to list the URI as an exception in your app/Http/MiddlewareVerifyCsrfToken.php middleware:

<?php

namespace AppHttpMiddleware;

use IlluminateFoundationHttpMiddlewareVerifyCsrfToken as BaseVerifier;

class VerifyCsrfToken extends BaseVerifier
{
    /**
     * The URIs that should be excluded from CSRF verification.
     *
     * @var array
     */
    protected $except = [
        'filepreviews/webhook'
    ];
}

The FilePreviews webhook controller we registered fires two events: filepreviews.success and filepreviews.error. Let’s create a Listener(app/Listeners/FilePreviewsSuccess.php) to handle the filepreviews.success event.

<?php

namespace AppListeners;

use Event;

use AppDocument;

class FilePreviewsSuccess
{
    /**
     * Handle the event.
     *
     * @param  array  $results
     * @return void
     */
    public function handle($results)
    {
        $document_id = $results['user_data']['document_id'];
        $document = Document::find($document_id);
        $document->preview_url = $results['preview']['url'];
        $document->preview = json_encode($results);
        $document->save();
    }
}

Let’s setup the event listener mappings for this application in app/Providers/EventServiceProvider.php.


protected $listen = [
        'filepreviews.success' => [
            'AppListenersFilePreviewsSuccess'
    ]
];

Our listener will react to the filepreviews.success event, look for a Document that matches document_id and update it.

To try this out locally I recommend using something like ngrok.

With php artisan serve running, on another tab/window run ngrok localhost:8000. Setup ngrok’s forwarding URL as your Callback URL on your FilePreviews application’s settings. Once that’s done navigate to http://localhost:8000/documents/create, pick a file, and submit. In a few seconds you should see a request logged in ngrok with the status 200 OK. If you then navigate to http://localhost:8000/documents you should see a link “Preview” instead of “No Preview”. Click that and you’ll see an image of the file you previously uploaded. Super cool stuff, right?

Bonus: Realtime with Pusher

Just because I’m really liking Laravel, I’ll show you how to add some realtime goodness using Pusher to what we already have working. Once we’re done we won’t need to refresh to see whenever our documents get previews.

Let’s add support for broadcasting events to Pusher.

$ composer require pusher/pusher-php-server

Go signup for a free account on Pusher, create an app, and set credentials in your project’s .env.

PUSHER_KEY=YOUR_PUSHER_APP_KEY
PUSHER_SECRET=YOUR_PUSHER_APP_SECRET
PUSHER_APP_ID=YOUR_PUSHER_APP_ID

Let’s create an event app/Events/FilePreviewsGenerated.php. This event will broadcast on the filepreviews channel an event called filepreviews.generated

<?php

namespace AppEvents;

use IlluminateQueueSerializesModels;
use IlluminateContractsBroadcastingShouldBroadcast;

use AppEventsEvent;
use AppDocument;

class FilePreviewsGenerated extends Event implements ShouldBroadcast
{
    use SerializesModels;

    public $document;

    /**
     * Create a new event instance.
     *
     * @return void
     */
    public function __construct(Document $document)
    {
        $this->document = $document;
    }

    /**
     * Get the channels the event should be broadcast on.
     *
     * @return array
     */
    public function broadcastOn()
    {
        return ['filepreviews'];
    }

    /**
     * Get the broadcast event name.
     *
     * @return string
     */
    public function broadcastAs()
    {
        return 'filepreviews.generated';
    }
}

We’ll tweak our FilePreviewsSuccess.php listener to fire a FilePreviewsGenerated event after saving our document.

<?php

namespace AppListeners;

use Event;

use AppDocument;
use AppEventsFilePreviewsGenerated;

class FilePreviewsSuccess
{
    /**
     * Handle the event.
     *
     * @param  array  $results
     * @return void
     */
    public function handle($results)
    {
        $document_id = $results['user_data']['document_id'];
        $document = Document::find($document_id);
        $document->preview_url = $results['preview']['url'];
        $document->preview = json_encode($results);
        $document->save();

        Event::fire(new FilePreviewsGenerated($document));
    }
}

All that’s left is add the Pusher JavaScript code on our index view.

<html>
<head></head>
<body>
  <h1>Documents</h1>
  <p><a href="{{ route('documents.create') }}">Create Document</a></p>
  @foreach ($documents as $document)
    <ul id="document-{{ $document->id }}">
      <li>ID: {{ $document->id }}</li>
      <li>Name: <a href="{{ $document->url }}">{{ $document->name }}</a></li>
      <li class="preview-url">
        @if ($document->preview_url)
            <a href="{{ $document->preview_url }}">Preview</a>
        @else
          No Preview
        @endif
      </li>
    </ul>
  @endforeach

  <script src="https://code.jquery.com/jquery-1.11.3.min.js"></script>
  <script src="https://js.pusher.com/3.0/pusher.min.js"></script>
  <script>
    $(function() {
      // Enable pusher logging - don't include this in production
      Pusher.log = function(message) {
        if (window.console && window.console.log) {
          window.console.log(message);
        }
      };

      var pusherKey = '{{ config('broadcasting.connections.pusher.key') }}',
          pusher = new Pusher(pusherKey, { encrypted: true }),
          channel = pusher.subscribe('filepreviews');

      channel.bind('filepreviews.generated', function(data) {
        var $doc = $('#document-' + data.document.id),
            $previewUrl = $doc.find('.preview-url');

        $previewUrl.html('<a href="' + data.document.preview_url + '">Preview</a>');
      });
    });
  </script>
</body>
</html>

With php artisan serve and ngrok running, we’ll need a new tab/window to run php artisan queue:listen.

The complete source code for this example project is available on GitHub.

If you have any questions or feedback on this post or FilePreviews.io, feel free to let me know.

Touch Typing and Programming

image

Programmers solve problems and most of the time use a keyboard to do so.

I firmly believe you need to be able to efficiently express yourself through typing. There’s probably nothing more fundamental in programming than being able to think of something and expressing it in code without losing your train of thought.

I say efficiently and not perfectly. You don’t have to be the fastest typist in the  west but speed helps. I could make up a couple of “imagine a surgeon that didn’t know the way around a scalpel” or “a baseball player that didn’t know how to swing” but I wont, you probably get the picture.

I was lucky enough to take typing lessons back in computer class in school. They gave use those orange covers you placed on top of the keyboard to hide all the keys. We’d use this with Mavis Beacon Teaches Typing and some other variation as a fun game. While most people lifted up the cover while the teacher wasn’t looking I just simply dealt with the damn thing and learned how to type instead of hunt and peck typing my way through it.

Before trying to become a great programmer, I suggest spending some time practicing, after all the only way to become a touch typist is through typing quite a lot. Check out: Typing.io, Mavis Beacon Teaches Typing for MacTyping.lk

Next you time you go to a programming workshop, you’ll be able to look at whatever the instructor is doing up front while you type, keeping up with everyone just fine.

Juice Box

image

A month ago I decided to take another stab at proposing a solution to allow all of us to keep working on great programming workshops.

Personally, the biggest headache in each and every one of the workshops that I have organized or participated in is the time and effort it takes to configure and setup the attendees’ computers. This results in attendees and instructors loosing valuable time that can be spent actually sharing knowledge. Various other people have shared the same worries and feelings about this, so might as well try and do something about it.


Juice Box
is virtual machine with a common environment that every workshop attendee can download and use before ever getting to the actual workshop.

What’s installed by default in Juice Box

  • Git
  • Python 2.7 with pip, virtualenv, and virtualenvwrapper
  • io.js with nvm
  • MongoDB
  • Redis
  • PostgreSQL
  • Docker
  • Sublime Text 3
  • Google Chrome
  • Firefox

What this project is trying to accomplish

  • Provide the most simple instructions possible so that attendees at every level can get started easily.
  • Reduce the need of a great network connection during the event.
  • Guarantee that all attendees have the same environment, tools and applications, making the process easier for all.
  • Reuse the same virtual machine between different groups and workshops.

Distribution

We built OVA files for Juice Box which allows us to simply distribute one file that contains the virtual machine. This file can be imported to VirtualBox with few clicks. We’ll be building and uploading each release so attendees can just download the resulting OVA file.  We’ve optimized the size of this file to achieve the smallest size possible but I’m sure it could be better. It’s smaller than an MP4 movie “shared” on TPB. Organizers might now require attendees to bring their Juice Box already downloaded and imported. Obviously, we might still have users that couldn’t download it, so just have a few USB flash drives with VirtualBox downloads for different operating systems and Juice Box’s OVA file.

InstallFests

Technical workshops shouldn’t be about installing a development environment in Linux, Windows, OS X, or whatever else. Unless that’s what they are about, this takes way to much time from everyone and it’s a pain. I propose that we organize InstallFests, events that are specific to helping users install development environments into their preferred operating system.

Open Source

Juice Box is completely open source under The MIT License and everything used to create the virtual machine is available on GitHub. If you’re a workshop organizer, make sure you “watch” the repository, this way you’ll get notified of new releases.

Feedback

Go download Juice Box and try it out out. This project might not be the best or correct solution, but it’s a start. Thanks to everyone that has already provided feedback. I’d love keep the conversation going and to have more feedback on this, wether you’re a workshop organizer or usually an attendee.

Sponsor

This project is sponsored by Blimp, a software products design and development agency. Blimp also organizes community workshops like Ember.js Puerto Rico and private trainings.

Logo by Jomarie Alvelo. Website and design by Giovanni Collazo.

Update

Just released v1.2.0! Juice Box now has support for Ruby with RVM.

Mako & Oliver

Three weeks ago today, on an unfortunate accident, Mako my 6 year old Siberian Husky escaped from home, and we noticed just a couple of hours later. After endless hours, driving around different possible routes, talking to people, calling out his name, posting hundreds of flyers, hundreds of people sharing on Facebook and Twitter, we haven’t found him yet.

Mako is more than a dog or pet to me, he’s a loyal friend, and we’ve been through a lot. This past weeks have been eternally tough for us. We’re still looking for him every day, we’re still receiving calls of people who’ve seen similar dogs around, I drive with my windows down in hope of hearing that beautiful distinct howl. We’re hopeful that perhaps a loving and caring family found him, rescued him, and has fallen in love with him. But we’re still not giving up.

While our search continues, we’ve brought along a new addition to the family to help. He’s not a substitute or replacement, nor is he a way to forget. Ana thought he’d be of great company, and surprised me with him. She was right and I’m so glad she did. We’ve name him Oliver. He’s a beautiful, happy, and intelligent 2 month old Miniature Dachshund. We’ve been together for two weeks now, he’s kept us company, brought us happiness and laughter, has helped make everything less bitter. He’s saved us and I love him so much.

In midst of it all, the people that really care about us have shined. I’ll be eternally grateful for everyone that showed support one way or another. And you, Ana, I love you so much, thank you for holding my hand along the way.

PS: Ana shared with me some helping points for when things don’t go your way. Might help you out some day too.

  • Take a step back and evaluate
  • Vent if you have to, but don’t linger on the problem
  • Realize there are others out there facing this too
  • Process your thoughts/emotions
  • Acknowledge your thoughts
  • Give yourself a break
  • Uncover what you’re really upset about
  • See this as an obstacle to be overcome
  • Analyze the situation – Focus on actionable steps
  • Identify how it occurred (so it won’t occur again next time)
  • Realize the situation can be a lot worse
  • Do your best, but don’t kill yourself over it
  • Pick out the learning points from the encounter

image

Love you bud.

Top 20 Hacker and Designer News 2014

Last year around this same date I started The Hacker and Designer News newsletter as a weekly curated recap of the what I thought were the best articles from Hacker News and Designer News on startups, entrepreneurship, hacks, programming, design, etc. I built and open sourced an aggregator to help me collect posts. I later built news.hdn.io using that same aggregator to view the current top posts on both sites side-by-side.

Learned a couple of things from building a newsletter, missed a couple along the way, but definitely looking forward to ramp up subscribers throughout 2015. If you still haven’t, feel free to check out past issues and subscribe.

Check out below the Top 20 Hacker and Designer News for 2014.

The Hacker and Designer News