Developing an App with ARKit & ARCore — The ChallengesARKit & ARCore - The challenges of developing an augmented reality app
We’ve been challenged by SIGVARIS with the idea of creating a proof of concept (POC) app that crossed educational content with augmented reality (AR). The objective was to experiment a new technology while providing users a better understanding of the benefits of its products.
SIGVARIS has its headquarters in Switzerland and is the global market leader in the manufacture of medical compression garments. It employs over 1,550 people and has its own production plants in Switzerland, France, Poland, USA and Brazil. SIGVARIS has subsidiaries in more than 70 countries on all continents. The Group pursues an international growth strategy on the solid foundation of more than 150 years of success.
The final POC consists of an app that contains information about the main SIGVARIS products and the benefits they have against venous related diseases.
Besides the usual text and video based information, using the device’s camera, the user can place a 3D leg on a flat surface and walk around it to observe the anatomy and tap on the veins to isolate them in the model and know the vein’s name.
A lot more can and will be done in future releases to improve the user’s understanding of the leg’s anatomy, related diseases and the positive impact that SIGVARIS products have on the well being of the user.
“ARKit blends digital objects and information with the environment around you, taking apps far beyond the screen and freeing them to interact with the real world in entirely new ways.” from the ARKit website
When it comes to programming for iOS based devices we have learned through experience that the closer you are to the ecosystem the more you can benefit from the current device’s capabilities and future technology releases from Apple. Besides the features announced for ARKit, this was another major decisive factor that we took into account when we decided to use ARKit, but you know how it is in the developing world, nothing comes without challenges:
We started the project with a super high-quality 3D model that looked really great on the device, but when we tried to submit the app to the store we got a warning saying that our app exceeded the size limit allowed.
Apple offers a couple of possible solutions for the problem like “On-demand resources”: “On-demand resources are resources (such as images and sounds) that you can tag with keywords and request in groups, by tag. The App Store hosts the resources on Apple servers and manages the downloads for you. The App Store also slices on-demand resources, further optimizing variants of the app.”
You can also reduce the quality of your model (i.e. number of polygons), in our case we could reduce the overall size for less than a half of its initial size without degrading the perceived visual quality.
ARKit can run on devices as old as the iPhone 6S which, according to our tests and to what the community reports, only allows apps to use as much as around 1GB of memory. Yet again if you have a very detailed model that takes a huge amount of space and consumes lots of memory to be rendered, the app might not be able to load the model without running out of memory and crashing.
A possible solution for this would be to reduce the size of your model or to have different models with varying quality for different devices.
SceneKit knowledge required
ARKit is built on top of SceneKit, which means that you won’t get away without having some basic knowledge about SceneKit. Do not get us wrong, we think this was a great architecture decision by Apple, is just that you won’t be able to do much without spending some time learning at least the basics of SceneKit.
As usual Apple did a great job creating a relatively simple API around such a complex topic like augmented reality.
The sample projects provided are a great way to start as they contain the full setup of an augmented reality app with different use cases for the technology. There is also a Github repository that contains multiple resources and projects that use the technology.
A great architecture decision was to create ARKit as an addition to the already existing 3D technology SceneKit. If you are a newcomer to Scenekit it will create an overhead at start but it will pay off as soon as you start interacting with the model with code.
ARCore is Google’s answer to Apple’s ARKit. It is designed to bring augmented reality capabilities to all Android phones in a relatively simple way. It’s also replacing the Project Tango, which required specialized hardware to run.
With all this hype around AR, Google couldn’t stay behind and it really wants you to see what ARCore can do (check this blog post). It’s awesome right? But is it really that simple to develop something like that? While developing this project to SIGVARIS we’ve found the answer: no it’s not simple.
ARCore project is just starting
If you go the ARCore website, you can see right away that the SDK is still in Developer Preview 2, and that it just started months ago. We’ve started developing our project when ARCore still was in Developer Preview 1, and with just one update the SDK changed almost entirely.
Lack of tools
If you’ve used ARKit or saw someone developing with it, you know that it has a cool interface, you kind of just need to drag a model into it and apply some textures and add some lights by clicking on some buttons, build the app into your phone, point the camera into a surface and boom, the model appears just like that. With ARCore, using the Android SDK, you have to code everything, and I mean: Every. Single. Thing. You want to load an OBJ into the scene? You must develop something to parse the OBJ yourself or use a third party library. Just keep in mind that using third party libraries for this can cause problems with ARCore if you’re not careful.
Lack of documentation
You want to know what a method does or check some guide to help you starting? Good luck with that. You just have a simple API reference which shows you a generic description for every class/method and a small guide that redirects you to the training page of the Android Developers website. At least you have a sample app that you can find here and use it as the base of your project.
Small list of compatible devices
If you read some articles about ARCore you easily understand that Google wants to bring it to every Android phone possible, but for now it’s only supported by a few devices. All variants of the Google Pixel 1 and 2 and some models of the Samsung Galaxy S8 to be more precise. So if you don’t have one to develop your next AR app for Android you’re screwed.
Our experience with ARCore was, at some points, frustrating, but it was also a rewarding one.
As you can now understand, our experience with ARCore was, at some points, frustrating to say the least, but it was also a rewarding one. You just can’t help but to feel proud seeing your 3D models using AR, it feels great.
So if you’re trying to develop something with ARCore, don’t give up. This might be a bad time to start the project of your dreams, but if you do so, you are forced to understand how everything works more deeply, and that can be a good thing for you since AR will open an all new window of opportunities in the future given the importance it’s gaining. Just don’t forget to help the community with your knowledge or to report issues here if you find bugs or want to see something improved.
Thank you for reading!
In case you don’t know, Coletiv is a software development studio from Porto specialised in Elixir, iOS, and Android app development. But we do all kinds of stuff. We take care of UX/UI design, web development, and even security for you.