Phoenix with image upload to S3 in an API: Implementation and testing

Here it is, another small article about the implementation of a new functionality in the API service we are building using Elixir and Phoenix.

Implementation

We need to implement a way to upload user images to the cloud (S3 in this case) so after some research I found this article that explains in a very good way the basic code on how to implement it. Arc is a very good dependency to handle image upload either for local or for S3 storage.

Store and fetch

After installing all the dependencies mentioned on the article, you can have a look into a simple controller, in this case I only specify the index and create methods, but all other methods (for example update) are similar.

defmodule MyApp.Web.AvatarController do ... def index(conn) do # We use Guardian to fetch user information current_user = current_resource(conn) if current_user != nil do image_url = Avatar.url({"image.jpg", current_user}) conn |> Phoenix.Controller.redirect(external: image_url) else send_resp(conn, :not_authorized, "") end end def create(conn, %{"avatar" => avatar}) do current_user = current_resource(conn) if avatar != nil do case Avatar.store({avatar, user}) do {:ok, _file_name} -> send_resp(conn, :ok, "") _ -> send_resp(conn, :service_unavailable, "") end else send_resp(conn, :bad_request, "") end end end

In this example, we start our methods by getting the user object. In this case we use Guardian, but you can get it from the database using Ecto. In our Avatar module generated by Arc using mix arc.g avatar we decided to incorporate the user ID into the file name.

defmodule MyApp.Avatar do ... # We use this so other users can't check other user profile images def filename(version, {_file, scope}) do :crypto.hash(:sha256, "a_very_long_string_#{scope.id}_#{version}") |> Base.encode16 |> String.downcase end # Override the storage directory: def storage_dir(_version, {_file, scope}) do "upload/user/avatars/#{scope.id}" end ... end

Note that #{scope.id} will use the user.id field, be sure you have it or change it to the field that you want to use.

After that, we check if the request has the avatar parameter, if not return a bad request HTTP status. If the parameter is present, we try to store it on S3 and we can check if it was successfully stored, checking the match pattern {:ok, filename}, in this case we could use {:ok, _} because we won’t use the filename.

Integration with Ecto

To use Arc with Ecto, you also need to add arc_ecto dependency. To add the field in the user table schema you need to start by creating a new migration to insert a new field.

# new_migration.exs defmodule MyApp.Repo.Migrations.NewMigration do use Ecto.Migration def change do alter table(:user) do add :avatar, :string end end end

After that, you need to add a new field into your user model and use the cast_attachments to validate the image upload and store the necessary information into the database.

# user.ex defmodule MyApp.User do use MyApp.Web, :model use Arc.Ecto.Schema schema "user" do ... field :image, MyApp.Avatar.Type end def changeset(user, params \\ %{}) do user |> Ecto.Changeset.cast(params, ...) |> cast_attachments(params, [:image]) end end

Be aware that when insert a new user the user.id isn’t available until the object is inserted on the database. You can generate an UUID (for example time + random number) to be associated to the filename or you need to insert the values first and then execute an update with the image only.

To view the image URL on the JSON structure I’ve created a simple method to correctly display the URL.

defmodule MyApp.Web.UserView do ... def render("show.json", %{user: user}) do ‰{ "id": user.id, "username": user.username, "avatar": render_image_url(user) } end def render_image_url(user) do if user.avatar != nil do Avatar.url({user.avatar.file_name, user}, :original) else nil end end end

To test if it’s working you can try it using Postman to send a POST request with form-data selected on the body and selecting a image file to upload.

Testing

Writing tests

In this example, I am going to write a simple upload test and check if it was successful or not.

defmodule MyApp.Web.AvatarControllerTest do ... test "Uploading test", %{user: user} do upload = %Plug.Upload{path: "test/assets/user_avatar.jpg", filename: "user_avatar.jpg"} post_params = %{"avatar" => upload} conn = conn_build() |> post(avatar_path(conn_build(), :create), post_params) # In this case we send in the response the location URL of the image assert List.first(get_resp_header(conn, "location")) == Avatar.url({user.id, user}) assert conn.status == 302 end

In this test, we select an image from our test assets and upload it using a POST method. We get the response, in this case we check for 302 (redirect) because we return the final url into S3 storage.

In this second test I’m going to show a test using Ecto (and ExMachina to build the model).

defmodule MyApp.Web.UserTest do ... test "Image Upload with Ecto" do {:ok, avatar_struct} = MyApp.Avatar.Type.load("x.jpg?1234567") user = build(:user, image: avatar_struct) # Be sure the image is available in the test folder upload = %Plug.Upload{path: "test/assets/avatar_user_1.jpg", filename: "avatar_user_1.jpg"} # POST parameters post_parameters = %{ "avatar" => upload } conn = build_conn() |> post(user_path(build_conn(), :create), post_params) json_response(conn, :ok) == render_json("show.json", user: user) end defp **render_json**(template, assigns) do assigns = Map.new(assigns) MyApp.Web.UserView.render(template, assigns) |> Poison.encode! |> Poison.decode! end

Using Fake S3

In order to implement tests to assure the behaviour of the functionality to upload images to AWS S3, we decided to use Fake S3, a fake AWS S3 API that replies in the same way as the real one, so we can test it at will without incurring in extra expenses and without an internet connection.

The readme file is pretty straight forward, and only two command lines are necessary to have it installed and running in your machine. After that you can check the code samples for different languages, in this case we are interested in Elixir. You can modify it on your config/test.exs and check if the port is the same as used in the fake server.

After that ensure that the Fake S3 server is running with the following command fakes3 -r ~/.s3bucket -p 4567, before you perform the tests. Run mix test and check if everything is as expected.

Running in Travis

To add this to your continuous delivery pipeline you can add the following lines into before_script, for example:

before_script: - gem install fakes3 - fakes3 -r $HOME/.s3bucket -p 4567 &

To not let the service hanging we can kill it after the tests finished running.

after_script: — kill $(pgrep -f fakes3)