FSM: Yea, I know

[schnee]

The FSM is down – I commited involuntary server-cide this weekend and I really don’t know when it will be back. You can go there, but the static reports likely won’t update and the portfolio and contest listings ain’t working. But I’m working on it.

In the meantime, here’s a random image for your enjoyment:

http://www.schneeworld.com/fsm/randomThumbnail.do

sorry.

[Elsinore]

Eep! I see you added the random images to the top of the FSM–that’s sweet! Hope that isn’t what killed it! Thanks as always for everything you do with the FSM–it really adds a lot to Farktography.

[schnee]

Good news and bad news.

Bad news is that the reports still are not working, but I’m slowly getting to the bottom of it. If you try to view a Participation Portfolio, you won’t get a null pointer exception anymore, so that’s progress. You won’t get anything useful, but…

Good news is that the “extract, transform and load” side of the FSM is working just fine. The HDR data is being extracted from Fark.com, stats and thumbnails are being generated and everything is being stored in the DB. So the data is intact.

[schnee]

I’m getting closer – the Participation Profiles are back (and slightly improved!) and now it is a “simple” matter of making sure everything else is too.

Doubtful for tonight, but maybe I can squeeze some time in in the next week.

Head-scratcher, that one was.

[Elsinore]

Ooo! Bigger thumbnails–very nice! Thanks again for sorting things out!

[monkeybort]

i’ve said it before and i’m sure i’ll say it again – schnee, you da MAN!!!

[schnee]

It is not completely sorted out yet (try a contest contour or search for a farktographer that doesn’t exist for all sorts of chewy exception goodness), but the path is clear.

The recent troubles with the reports made me forget how hard it was to do the damn thumbnails. Java does not make it easy. Did you know that when you do an Affine Transform, an alpha channel gets introduced? That sucked.

One thing that I did learn is that the oft-repeated advice to resize in steps is very good advice. I originally built the thumbnails in one step (e.g. from 700px to 200px) and they didn’t look too good. Now, I do it in 10 steps and they look much better.

If you want to link to a thumbnail, you can “simply” do something like:

http://www.schneeworld.com/fsm/thumbnail.do?tnc={the code}

e.g. http://www.schneeworld.com/fsm/thumbnail.do?tnc=12345

You can find the code by looking at the source of a portfolio – the URL is right there. First person who tries to run through all 12461 entries gets slapped and then I’d have to do something like IP filtering. Not every entry has a thumbnail; such is the web.

I tried really hard to keep the image inside of a 200×200 pixel box, but sometimes it leaks out to 201. Zeke, Brian, feel free to stick them on FT.net, or use the random thumbnail URL – right now, that selects from the top 100 vote getters in the last 4 contests, but I’ll probably tune it a bit (suggestions?).

[Elsinore]

Ok, I have no idea what an Affine Transform is, but the fact that it intoduces an alpha channel is pretty wild. So did the thumbnails just not show up at that point?

[schnee]

Ok, I have no idea what an Affine Transform is, but the fact that it intoduces an alpha channel is pretty wild. So did the thumbnails just not show up at that point?

They showed up – the JPEGs got written out just fine (apparently, alphas are part of the JPEG standard) but Netscape and IE (and Photoshop) don’t know what to do with a 4-channel JPEG. They looked like someone scanned a transparency as a color-reversal.

Originally, I created PNG thumbnails. They displayed just fine, but were 6 or 7 times bigger than the JPEG equivalents.

Affine Transform is “the” way to resize. When you select “bicubic” or “nearest neighbor” or whatever when you resize, there’s a good chance that whatever tool you are using (GIMP?) is applying an Affine.

[Elsinore]

Ahhhh. That’s wild that alphas are part of JPEG standard. Anytime I forget to flatten an image with an alpha channel and try to save it off as JPG, GIMP complains and tells me it has to export if I really want to continue with my save.

Good info about Affine Transform, though. GIMP’s “best” resize interpolation is labelled “cubic” which I always figured was really bicubic. It does a really good job, though, whatever it is 😉 For the next version of GIMP coming down the line, they still have “cubic” but they’ve added “lanczos” as the best option. No clue what that is, though.

/huzzah for tangents!

[monkeybort]

schnee – i’m sure you know this, but it seems like the reports still aren’t working. i can get to my portfolio and whatnot but any of the reports i click on gives me a http 404 message saying the links aren’t available.

sorry if you already know this and are working on it!!!

[schnee]

“nearest neighbor”, “cubic”, “bicubic” and “lanczos” are all sampling algorithms used by Affine Transforms. When the FSM makes its thumbnails, it uses “bicubic”.

I’m fuzzy on the details, but when an Affine is applied, it maps from one space to another. If those spaces aren’t the same size (or shape), pixels are either discarded or created. Those pixels are defined by the sampling algorithm – “nearest neighbor” will just duplicate the pixel closest. The other algorithms do a better job at figuring out how to create or destroy pixels.

“Lanczos” is apparently all the rage amongst the cool kids. It is supposed to be quite good. When it becomes available to me, I’ll use it in the FSM.

(one of the raisons d’etre of the FSM is to allow me to monkey around with whatever captures my interest – Levenstein Distance Algorithm, Atom syndication, image processing, comparative statistics – all there. Someday, I may even allow someone to search for farktographers based on what their name sounds like)

[mikemikeb]

For the next version of GIMP coming down the line, they still have “cubic” but they’ve added “lanczos” as the best option. No clue what that is, though.

“Lanczos” is apparently all the rage amongst the cool kids. It is supposed to be quite good. When it becomes available to me, I’ll use it in the FSM.

Is there anything Wikipedia can’t do?

[schnee]

Oh, in case anyone was wondering how long it took me to do all the past thumbnails, here’s an entry from my log:

Processed 7218 in 5 hours 27 minutes 51.141 seconds (2.725289692435578 s/e)

7218 entries is not all of them, mind you, but the 2.7 seconds is the average time needed to fetch the image from the URL, resize it and stuff it into the DB. The big time waster in processing historical images is waiting for the HTTP GET timeouts for those images that no longer exist. Multithreading would have helped here but I didn’t want to mess with it.

For recent contests, images are thumbnailed on the fly at a rate of less than a second per. This is due to image URLs being mostly valid for fresh entries.

I also SHA-1 the original image bytes, which was suggested by Zeke as a possible indicator of Recycled entries. I haven’t done anything with that yet.

[Elsinore]

Wow 5.5 hrs to process? damn…

[Author]

Posts