This was bugging me over the weekend: What is a good way to solve those *Where's Waldo?* [*'Wally'* outside of North America] puzzles, using Mathematica (image-processing and other functionality)?

Here is what I have so far, a function which reduces the visual complexity a little bit by dimming some of the non-red colors:

```
whereIsWaldo[url_] := Module[{waldo, waldo2, waldoMask},
waldo = Import[url];
waldo2 = Image[ImageData[
waldo] /. {{r_, g_, b_} /;
Not[r > .7 && g < .3 && b < .3] :> {0, 0,
0}, {r_, g_, b_} /; (r > .7 && g < .3 && b < .3) :> {1, 1,
1}}];
waldoMask = Closing[waldo2, 4];
ImageCompose[waldo, {waldoMask, .5}]
]
```

And an example of a URL where this 'works':

```
whereIsWaldo["http://www.findwaldo.com/fankit/graphics/IntlManOfLiterature/Scenes/DepartmentStore.jpg"]
```

(Waldo is by the cash register):

I've found Waldo!

**How I've done it**

First, I'm filtering out all colours that aren't red

```
waldo = Import["http://www.findwaldo.com/fankit/graphics/IntlManOfLiterature/Scenes/DepartmentStore.jpg"];
red = Fold[ImageSubtract, #[[1]], Rest[#]] &@ColorSeparate[waldo];
```

Next, I'm calculating the correlation of this image with a simple black and white pattern to find the red and white transitions in the shirt.

```
corr = ImageCorrelate[red,
Image@Join[ConstantArray[1, {2, 4}], ConstantArray[0, {2, 4}]],
NormalizedSquaredEuclideanDistance];
```

I use `Binarize`

to pick out the pixels in the image with a sufficiently high correlation and draw white circle around them to emphasize them using `Dilation`

```
pos = Dilation[ColorNegate[Binarize[corr, .12]], DiskMatrix[30]];
```

I had to play around a little with the level. If the level is too high, too many false positives are picked out.

Finally I'm combining this result with the original image to get the result above

```
found = ImageMultiply[waldo, ImageAdd[ColorConvert[pos, "GrayLevel"], .5]]
```

My guess at a "bulletproof way to do this" (think CIA finding Waldo in any satellite image any time, not just a single image without competing elements, like striped shirts)... I would train a Boltzmann machine on many images of Waldo - all variations of him sitting, standing, occluded etc; shirt, hat, camera, and all the works. You don't need a large corpus of Waldos (maybe 3-5 will be enough), but the more the better.

This will assign clouds of probabilities to various elements occurring in whatever the correct arrangement, and then establish (via segmentation) what an average object size is, fragment the source image into cells of objects which most resemble individual people (considering possible occlusions and pose changes), but since Waldo pictures usually include a LOT of people at about the same scale, this should be a very easy task, then feed these segments of the pre-trained Boltzmann machine. It will give you probability of each one being Waldo. Take one with the highest probability.

This is how OCR, ZIP code readers, and strokeless handwriting recognition work today. Basically you know the answer is there, you know more or less what it should look like, and everything else may have common elements, but is definitely "not it", so you don't bother with the "not it"s, you just look of the likelihood of "it" among all possible "it"s you've seen before" (in ZIP codes for example, you'd train BM for just 1s, just 2s, just 3s, etc, then feed each digit to each machine, and pick one that has most confidence). This works a lot better than a single neural network learning features of all numbers.

I don't know Mathematica . . . too bad. But I like the answer above, for the most part.

Still there is a major flaw in relying on the stripes *alone* to glean the answer (I personally don't have a problem with *one* manual adjustment). There is an example (listed by Brett Champion, here) presented which shows that they, at times, break up the shirt pattern. So then it becomes a more complex pattern.

I would try an approach of shape id and colors, along with spacial relations. Much like face recognition, you could look for geometric patterns at certain ratios from each other. The caveat is that usually one or more of those shapes is occluded.

Get a white balance on the image, and red a red balance from the image. I believe Waldo is always the same value/hue, but the image may be from a scan, or a bad copy. Then always refer to an array of the colors that Waldo actually is: red, white, dark brown, blue, peach, {shoe color}.

There is a shirt pattern, and also the pants, glasses, hair, face, shoes and hat that define Waldo. Also, relative to other people in the image, Waldo is on the skinny side.

So, find random people to obtain an the height of people in this pic. Measure the average height of a bunch of things at random points in the image (a simple outline will produce quite a few individual people). If each thing is not within some standard deviation from each other, they are ignored for now. Compare the average of heights to the image's height. If the ratio is too great (e.g., 1:2, 1:4, or similarly close), then try again. Run it 10(?) of times to make sure that the samples are all pretty close together, excluding any average that is outside some standard deviation. Possible in Mathematica?

This is your Waldo size. Walso is skinny, so you are looking for something 5:1 or 6:1 (or whatever) ht:wd. However, this is not sufficient. If Waldo is partially hidden, the height could change. So, you are looking for a block of red-white that ~2:1. But there has to be more indicators.

- Waldo has glasses. Search for two circles 0.5:1 above the red-white.
- Blue pants. Any amount of blue at the same width within any distance between the end of the red-white and the distance to his feet. Note that he wears his shirt short, so the feet are not too close.
- The hat. Red-white any distance up to twice the top of his head. Note that it must have dark hair below, and probably glasses.
- Long sleeves. red-white at some angle from the main red-white.
- Dark hair.
- Shoe color. I don't know the color.

Any of those could apply. These are also negative checks against similar people in the pic -- e.g., #2 negates wearing a red-white apron (too close to shoes), #5 eliminates light colored hair. Also, shape is only one indicator for each of these tests . . . color alone within the specified distance can give good results.

This will narrow down the areas to process.

Storing these results will produce a set of areas that *should* have Waldo in it. Exclude all other areas (e.g., for each area, select a circle twice as big as the average person size), and then run the process that @Heike laid out with removing all but red, and so on.

Any thoughts on how to code this?

Edit:

Thoughts on how to code this . . . exclude all areas but Waldo red, skeletonize the red areas, and prune them down to a single point. Do the same for Waldo hair brown, Waldo pants blue, Waldo shoe color. For Waldo skin color, exclude, then find the outline.

Next, exclude non-red, dilate (a lot) all the red areas, then skeletonize and prune. This part will give a list of possible Waldo center points. This will be the marker to compare all other Waldo color sections to.

From here, using the skeletonized red areas (not the dilated ones), count the lines in each area. If there is the correct number (four, right?), this is certainly a possible area. If not, I guess just exclude it (as being a Waldo center . . . it may still be his hat).

Then check if there is a face shape above, a hair point above, pants point below, shoe points below, and so on.

No code yet -- still reading the docs.

I agree with @GregoryKlopper that the *right* way to solve the general problem of finding Waldo (or any object of interest) in an arbitrary image would be to train a supervised machine learning classifier. Using many positive and negative labeled examples, an algorithm such as Support Vector Machine, Boosted Decision Stump or Boltzmann Machine could likely be trained to achieve high accuracy on this problem. Mathematica even includes these algorithms in its Machine Learning Framework.

The two challenges with training a Waldo classifier would be:

- Determining the right image feature transform. This is where @Heike's answer would be useful: a red filter and a stripped pattern detector (e.g., wavelet or DCT decomposition) would be a good way to turn raw pixels into a format that the classification algorithm could learn from. A block-based decomposition that assesses all subsections of the image would also be required ... but this is made easier by the fact that Waldo is a) always roughly the same size and b) always present exactly once in each image.
- Obtaining enough training examples. SVMs work best with at least 100 examples of each class. Commercial applications of boosting (e.g., the face-focusing in digital cameras) are trained on millions of positive and negative examples.

A quick Google image search turns up some good data -- I'm going to have a go at collecting some training examples and coding this up right now!

However, even a machine learning approach (or the rule-based approach suggested by @iND) will struggle for an image like the Land of Waldos!

Similar Questions

ref^ refers to the commit before ref, what about getting the commit after ref? For example, if I git checkout 12345 how do I check out the next commit? Thanks. PS Yes, git's a DAG node pointer struct

If I have an option, for example AxesStyle, how can I find possible values for it? I know that one of the possible values is Arrowheads, but can I find others? I was looking for lists or something lik

How do I programatically find what java keystore file my JVM's default TrustManager is using? This question is similar, but the answers are all actually about which keystore file it should be using.

I need to write a function to find the mode of a array. I'm not good at coming up with algorithms however and I'm hoping someone else knows how to do this. I know the size of the array and the values

This question already has an answer here: Using Git how do I find changes between local and remote 9 answers I'm working in a branch that is remote tracking. I've made several commits. Is there

How do I determine the module for a given cmdlet for direct calling from a function that overrides the cmdlet. For example, how am I supposed to find out that Start-Transcript lives in Microsoft.Power

I am trying to plot a function in Mathematica that is defined over the unit simplex. To take a random example, suppose I want to plot sin(x1*x2*x3) over all x1, x2, x3 such that x1, x2, x3 >= 0 and

I was wondering the other day if StackOverflow had an API I could access from Mathematica, and apparently it does: Saving plot annotations What's the best way to get data from StackOverflow into Mat

I know the email from a mail enabled list, with I have no idea where the list lies. How do i find the list to which the email belongs to?

How do I find out who the author of a revision is in SVN?

I have N number of scripts included in a webpage, how do i find out which script is declaring variable X or which script is implementing function Y() ? Kind of like linux's which/type root@mac:~$ whic

Mathematica seems to be missing a function for this, or I can't find it anyway. The Series function can do expansion in succession for multiple variables, but it doesn't seem capable of doing a full m

How do I find out all of the strings surrounded with double quote and replace them a new string? Input Output ----- ------ `Hello User, what's up?` -> `Hello l(User), what's up?` `Regexes are

How can I label each of these lines separately : Plot[{{5 + 2 x}, {6 + x}}, {x, 0, 10}]

How do I find what version of GTK I have installed on a Solaris box?

Is there something available that tells me what .NET version I am using and whether it is .NET 2.0 SP1? Thanks

I have a need to edit cue files in the first directory and not go recursively in the subdirectories. find(\&read_cue, $dir_source); sub read_cue { /\.cue$/ or return; my $fd = $File::Find::dir; my

I'm a java programmer, and if I see something that: I don't know about or just want to find a method description without opening an ide or am on support I type java [classname] into google, and the

How do I find out if a file is a photo or a video in Qt?

I have been reading about how symbolic languages works, focusing more on Mathematica. As far as I could understand, to evaluate is to apply a sequence of transformation rules to the input until no mor

I am going to launch my PHP/mySQL website. I have enabled statistics on mySQL. However, what tools can I use to find out the missing indices?

I’d like to enumerate all subfolders of a folder in Haskell. Getting all folder contents is easy, there’s agetDirectoryContents function. But how do I filter them? Since getDirectoryContents ret

I can't seem to find a specific node in the graph without traversing the whole thing. Is there something I'm missing? I'm using tinkerpop blueprints. Orientdb gives some sort of unsemantic id to a no

I am writing a Perl script that is iterating over file names in a directory and its sub-directories, using the following method: find(\&getFile, $mainDir); sub getFile { my $file_dir = $File::Find

I'm creating my own ActionSheet class and I need to emulate size and font UIActionSheet's text. How do I find size and font UIActionSheet's text?

How do I find the remixes of a given song using Echo Nest's Song API? I can't seem to find any documentation in the API methods page that indicates whether a song is a remix of anther song. Is there a

For example, str = 'abcdefg'. How do I find the index if c in this string using Ruby?

I'm sure this is an incredibly simple question but I've recently started using Mathematica so it's all new to me. How do you scale the y axis? I try the following but it won't compile. Plot[y = {Exp[-

I have a timestamp, say $now, I need to find the timestamp for, say, 45 days after the date of $now. What do I do?

I have an image which has been encrypted, and then hidden in a cover image. and finally retrieved. I'd like to find the horiz, vertical and diagonal correlation coeff's of the initial image, and the c

How do I find out what directory my console app is running in with C#?

LastRow = Cells.Find(*, [A1], , , xlByRows, xlPrevious).Row works very well for finding the last row in a worksheet. But how can I tweak the line above to find the last row of column C? I've tried

I Cannot find how to use relative paths in mathematica. My directory structure is simple. Import[G:\\Research\\Acc and Vel Runs\\5-24\\Mathematica\\Data\\250 \ Acc.xls][[1]] // TableForm That Demon

How can I find libraries which have newer versions with sbt. For Ruby bundler, it can be done with bundle outdated. I want to do similar thing with sbt.

I am trying out some CGI-scripting in Python. If a request come to my python script, how do I find the mime-type of the filetype? UPDATE: Adding more info Some images for now, adderess/image.jpg. But

How do I get to a Palette that's not in my Palettes menu? For instance, I occasionally need Notation palette, and it's not in that menu

How can I save the numerous Plot options in Mathematica into a single variable

Suppose we have two finite line segments defined each by two points (in two space). I would like to find a way to get the intersection point of those two lines. Eventually, I would like to extend this

How do I find the relevant model where its name equals something? I've tried: this.get('content').findProperty('name', name), but it doesn't appear to work. I suppose it'd work if my enumerable wasn't

Does anybody know if there is a built-in function in Mathematica for getting the lhs of downvalue rules (without any holding)? I know how to write the code to do it, but it seems basic enough for a bu

Q: How do I find the available PBS queues on the typical Torque MPI system? (asking our admin takes 24+ hours, and the system changes with constant migration) (for example, Std8 is one possible qu

I have no idea what prompted this, but now in VS, I have a horrible find/replace window that pops up and rarely even works. How do I get the normal one back?

Can Mathematica do Bayes Rule conditional probability calculations, without doing the calculation manually? If so how? I have been searching both the Mathemtaica doco and the web for a hint but cannot

As the question asks, how do i run a find/replace in pycharm for a word over the entire project?

How can I use Mathematica and Google scholar to find the number of papers a person published in 2011?

i need to find the complete path of a given Linux executeable? How can i do this? For example the real path of cat is /bin/cat

I am beginner in Mathematica, and can only understand basic command so far. I need help in writing the maximum likelihood estimation function. I want to find the MLE of a Binomial distribution, with 2

I am running Mathematica 7, and I am trying to run a simple Do loop in parallel, using ParallelDo. The following standard, sequential code works fine: len = 10; A = Table[0, {len}]; Do[ A[[i]] = i*10;

I'm not sure I got how to define private functions right. When I'm writing a package mathematica, I just do this: BeginPackage[myPackage`] myPublicFunction::usage=myPublicFunction blahblahblah; Be

Here's a somewhat general computer question. I've always been able to follow the LOGIC of programming, but when I go to code something, I always find that I don't know some method or another to get wh