Neural Network Zoom Experiments

 

 

Neural Networks as a tool to both zoom and enhance images

Here I will document my experiences using recurrent neural networks to supersample and enhance images. A properly trained neural network should be able to approximate a zoomed picture. It won’t be able to show information that was not visible before, but it is able to remove certain artifacts that other transformations can create.

Neural Networks use an input layer that consists of the image information, a series of hidden layers of various dimensions, and an output layer that is the corrected image. Neural Network layers consist of matrices containing gradient weights and biases that when multiplied with an activation function such as ReLU or Sigmoid will produce a simulation of a neuron firing.

Training this network of weights can be done via recursion. This calculates the error between the actual output and the expected output and uses it to train the neural network to minimize that error.

For our first example, I will take a very small circle with a picture of a Linux penguin on the inside. Our starting picture will be 100×100 and we are going to upsize it to 1000×1000 via both neural networks and traditional transformation mechanisms.

Different methods of resizing

I will be attempting multiple methods of both neural and traditional resizing and will compare their results as objectively as possible.

Traditional Resizing (linear filtering)

 

 

⇒ Traditional Algorithms

 

 

 

 

As a control, I will resize using paint.net using the default settings. This uses a series of algorithms to fill the empty pixels with a mixed gradient allowing for a somewhat smooth resize.

 

 

 

 

Attempts using Neural Networks

Attempt 1 – alexjc/Neural-Enhance

My first attempt at resizing this photograph was to use the python toolNeural-Enhance with Windows 10. It should have been simple, but it turned into a disaster. I was never able to figure out how to mount the filesystem correctly. All the documentation that I could find on this issue is for other operating systems and was specifically not applicable to windows. This is due to the way the Unix filesystem is structured.  I tried both forward, and

backslash, full path, relative path, mounting the directory and trying, all to no avail. It is able to load the model, but I am unable to load the image file that I want to enhance.

I have had this working before in Linux, and plan on looking back into it again in the future. When I do I will update this article.

Attempt 1 Results- Failed to enhance images. Will look into it in the future.

 

 

Attempt 2 – Waifu2x Tool (x4 times)

I had to use this 4 times in a row to achieve the desired result as it only supported supersampling by factor x2. I initially wanted to steer clear of tools like this, but it’s always nice to have all the available information.

Single-Image Super-Resolution for Anime-Style Art using Deep Convolutional Neural Networks. And it supports photo.

It seems to work on a similar neural network principle as Neural-Enhance. It would not surprise me if they share the same (TensorFlow?) roots.

Resized using Neural Networks via Waifu2x (4x)⇒

 

 

 

 

 

 

 

 

 

 

 

Attempt 2 Results: Promising

Attempt 2 gave promising results. There are noticeable artifacts present around the lower fins of the penguin, present in the form of yellow color bleed. This is an issue absent in normal resizing tools. Besides that the image is clear, the lines concise, and there is no noise, much unlike its traditional counterpart.

 

Attempt #3 – A Paid service (It’s not good)  https://letsenhance.io/

I signed up for their trial. Uploaded my 100×100 image, and clicked the “magic” button. Here was the result.

Attempt #3 Results: Very Poor

Very disappointing to say the least, there is no reason I can see to use this over the free alternatives. Note the loss of pattern between the checkers, and the general noise in the image. I was able to get much better results off the above Waitfu2x tool for free, with less effort. Perhaps this is some sort of sophisticated proof of concept that flies right over my head. Or maybe people do buy into this marketing hype. Obviously much better results can be had for free than using https://letsenhance.io/ or any other paid service.

Attempt #4  – TensorZoom(Android)

This one is certainly the most frustrating to test. There are builds that can run on Windows, but I was unable to get it working. I didn’t document the process, but there was nothing useful learned. Instead, I will be using my Android Google Pixel OG phone to magnify the photo. Unlike the other examples, there is no limit to TensorZooms zoom setting.

 

Attempt #4 Results: Outstanding.

 

Neural Networks seem to do an alright job of upscaling small images with minimal artifacts. Depending on the method used the resized version may be much clearer than its traditionally scaled counterpart. It seems like areas of struggle in the image was during transitions between the penguin and the background. This means that Neural Network image scaling may be more prone to artifacts when there is more noise (such as a crowd of people in an image)

 

 

 

This is an article in progress. Neural Networks stand as a promising way to resize images in the future. They are currently capable of outperforming traditional resizing in some cases.

These are pictures I took during the 2017 Thomas fire. The majority of them are within UCSB/Isla Vista.

 

Click to enlarge.

 

Pictures of the Thomas Fire that I took up close from an Amtrak train.

These are photos and videos of the Thomas Fire, as seen by an Amtrak Train that I recorded with a smartphone camera. These aren’t amazing quality, as they come from a phone camera sensor. However, as far as I am aware, this is the only video that exists of this fire through these remote areas along the train tracks.

 

Take a look at my post here: http://bijanmoradi.org/isla-vista-and-ucsb-during-the-2017-thomas-fire/ to see what the smoke and ash plume looks like inside UCSB 20 miles north west.

 

 

Pictures

 

During the fire along the coastal surfliner track.

The same area three days later from the same route

 

Videos