May 12, 2016

Algorithmically Determined, Mom Contested

I recently came across this new camera roll app for IOS that not only clusters photos based on subject matter, but computes an “ascetic score” and can show you your most ascetically pleasing photos. As someone who has 4,939 photos on her phone, and a love of machine learning, I had to give it a try.

My most aesthetically pleasing photo this month is… drumroll

nail
A nail sticking out of the floor which I have caught my foot on way too many times. I took this photo so I could show the kind folks at home depot what I was talking about. It just will not stay down no matter how many times I hammer it!

I disagree. I mean it’s a fine nail and all, but my most aesthetically pleasing photo? Adding insult to injury, the nail in the floor board photo only received an ascetic score of 71%.

The clustering results were mixed. Most of the time I’d get a several clusters of what I’d consider to be the same subject matter. Admittedly, it’s more handy to scroll through effectively ten photos rather than hundreds. In scrolling through the curated roll I found cute photos I had forgotten I had taken. But the other kind of fail, when photos are clustered that shouldn’t be, is much less forgivable. When the app fails, it fails big time.

Take the following example:

ac

bd

In the above photos, photo A was in a cluster with similar photos, while B, C, and D were clustered together. B & C I can understand; their different children but the same general framing, although A & C are closer in framing. I would prefer A & B to be a cluster, but wouldn’t mind A, B & C. And D? What the heck happened there? Also the respective image ascetic scores are 38%, 38%, 36% and 24%. While I wouldn’t use them to showcase my work, I do think they beat nail-in-floor-board. Except maybe photo D. I have no idea what I was going for there.

Me being me, I set out to reverse engineer the algorithm and see if I could figure out what is happening.

It appears that photos are being clustered based primarily on the timestamp. Cluster B, C and D’s photos were taken from 9:16 – 9:17, where as Cluster A’s photos were taken at 9:18. It appears that a photo is added to the current cluster if it’s tags are similar to the photos in that cluster, otherwise a new cluster is formed. There’s no interleaving. As a result, visually similar photos are sometimes buried distributed in different clusters. “Buried” might be a better term, if visually different photos have a higher ascetic score and are used as the key example from the cluster. For example, the top photo in B, C and D’s cluster is another photo of Alexis. This an undesirable feature is made extra bad by the fact that the app offers to delete “visually similar” (i.e. the less ascetically pleasing photos in the cluster.) That would mean all photos of Nicole by the flowers.

I can’t help but wonder how I would have clustered the photos. My first instinct was also to use the timestamp, however, I would allow for clusters to interleave. When taking photos I don’t decide to take photos of one child in the swing, switch to the other child on the slide, and then call it done. I usually go back and forth between the girls with my attention. Clearly Photos B & A should be in the same cluster, even though D was taken after B but before A.

My next instinct was to cluster photos based on color similarity. Putting the above four photos through my Image Color Pallet app I get the following pallets:

colorcomparephotos

Photos A & B have identical color pallets. Photo C is very similar, but has a blue cluster in place of the pink cluster, as the two girls are wearing different color jackets. Photo D, on the other hand, is very different. Clustering based on color pallets would put A & B in the same cluster, and possibly add photo C depending on the sensitivity threshold.

While aesthetics can be a bit subjective, I was delighted to find a few research papers on the topic of creating an algorithmic base approach to predicting how aesthetically pleasing an image is. This knowledge will come in handy for the photograph apps I’m working on!

Related posts:

Posted in Internet & Technology


Comments

  1. Hi, is there an similar app in Android?

  2. Hi, is there an similar app in Android?

    • Not that I know of. The app I used was the roll which doesn’t seem to have an andoid version.

Trackbacks


Leave a Reply

Your email address will not be published.