NFO file-based scanning is much slower than letting Boxee ID online

[RoyalEF]

Using XBMC I created NFO, TBN files for everything. I converted movies first--that went well.

Now I converted all the TV Shows (TVSHOW.NFO, EPISODE NFO & TBNs). Now, identifying is so slow. It's been at it for over 3 hours and it is only down (from Z) to Stargate. I'm probably looking at 11-14 hours of identifying compared to 5-8 hours before.

I wonder what is making the difference in the huge slowdown.
- amount of excess tags in TVSHOW & EPISODE.NFOs (XBMC exports far more than Boxee uses)
- hidden .actor folders and tbn files - my opinion is boxee should ignore hidden folder and files.
- afp-based tbns instead of online fetches
- three afp-based files per video, tripling the number of files to scan and ID

I've already restarted once just to see if that would help.

Has anyone else converted to NFO-based and notice a difference in scan/id times?

[Digitalist]

A couple of questions for clarification:

1) Do your .NFO files contain <ID> tags? If so, Boxee Box will look the shows up online regardless of what you have in your NFO file (as far as I know) so I make sure to remove <ID> tags from all the tvshow and movie NFO files.

2) What resolution are your folder.jpg and <episodename>.tbn files at? This can make a difference

3) What program did you use to produce the meta-data files?

4) From what you've written it seems you set BB to scan your entire media directory as a single file source, many people on this forum have reported *much* faster media scanning and identifying by adding one TV series at a time as file sources (also nice because you can set "Scan Once" for all completed series where you won't be adding episodes).

Just to relate my own experience, I have all of my media in the ideal folder/filename structure and created tvshow.nfo, folder.jp and <episodename>.nfo and tbn files using Ember Media Manager and once I sorted out a few corrupt thumbnails I found scanning and identifying to go *very* quickly. No more than 3 minutes for a 64 episode TV series, for example.

[RoyalEF]

Just got home, 7 hours later. TV SHows are about 70% done. Boxee was very slow to respond but did eventually react to each remote action. I menu'd a shutdown.

1) I changed the <ID></ID> to <ID-TVDB></ID-TVDB> in all the TVSHOW.NFO files -- this way I can reverse the change in case I wanted to revert to Boxee's method.

2) I got rid of folder.jpg because, in the past the presence of this seemed to negate the thumbnailing process per episode if a episode tbn didn't exist. When present it would display a chunk from the center of the folder.jpg--which was 100% useless a 100% of the time. Why they would EVERY choose to implement that behavior is beyond me. TVDB doesn't always have episode thumbnails. The tvshow thumbnail is pulling the first THUMB from TVSHOW.NFO

Every episode tbn seems to be 400x300. They are pulled by XBMC from TVDB.
(edit) Through syslogs, the boxee's copy stored internally appears to be nearly identical in size to the original TBN file. 12 byte difference on one, 19 byte difference on another.

The TVSHOW's TBN is dynamically pulling a poster sized from TVDB. I'm not sure if when Boxee pulls posters down normally whether they pull them from their server or from TVDB. This is a possible issue, since from their own servers I imagine they've been sized. I plan to create local posters as I'm trying to get my local library to operate without an internet connection--and still have the graphics and info.

The poster thumbnails I've checked have been 680px × 1000px.
(edit) Through the syslogs I found an example. TVDB's 680x1000 was 351kb. Boxee stored the image in special://masterprofile/profiles/myname/Thumbnails/Video/d/d3f0ffa7.tbn with a size of 61,199 bytes. So it is definitely resizing them down significantly. I figured it was doing that as I could see a difference in quality of the poster now, vs the boxee poster before. I'm thinking it doesn't do the greatest resize algorithm.


3) XBMC

4) I've contemplated that. I have three shares: Movies, TV Series and a test where I prove stuff out, get XBMC to ID the series then move the folder into it's permanent spot. This lets me do that add and replace easy without disturbing the larger shares. I set the big shares for scan once and manually choose if I need a rescan or remove/add. I'd hate to have to add 150 sources!

When I have only a dozen NFOs for problem shows the add went quite fast. I could literally see shows added every few minutes. It added several movies per minute.

I've checked syslogs and they actually look like it isn't doing much. I've turned on DEBUG logs. It generates a lot of stuff, but it will let me see each step it is taking.

Although the identifying: status on FILE SOURCES is remaining blank the CPU is 100% so it is doing something.

[RoyalEF]

I think I found at least part of the issue. Just noticed that my NAS is doing a RAID scrubbing. It started 6PM, several hours into the scan, which would make the situation even worse.

[RoyalEF]

Two days later and several hung reboots it was done, but I still have some bad posters and needed to remove/add source. I decided to wipe all the sources and just add one- movies. I noticed that I had added the sources back in with a user/pass access instead of read-only guest. I went back to that this time. After one or two forced reboots it is done. The scan found and Identified 1180 files. Unfortunately, only 390 movies were added and that number hasn't moved in the last hour. So Scanned, Identified, done & Idle. But I'm missing about 700 movies.

This issue started happening after I used the Wipe Media from database option for the first time. I did a second one and the problem seems worse. Previously I got through most of movies, and eventually through the TV share. I know the media info wipe and the FACTORY DEFAULT are common "I've no clue what is wrong, try this" options. Much like Microsoft telling us the only way to fix our problem is to wipe out your computer and re-install Windows. That's what I do when my house needs some repairs. I burned it down to the foundation and start over!!

I'm contemplating factory default, because I haven't been able to reliably add sources to the Boxee in about a week now. I'm thinking the db is trashed in someway. Hopefully factory default will truly wipeout the db and start from scratch.

It's my only guess. Either that or using all NFO-based IDs & TBNs for your movies & TVshows is a disaster and unusable. I'd think if that was the case someone other than me would have come to that conclusion first.

[RoyalEF]

Ok, i don't get it. I did the RESTORE to factory defaults, CLEAR MEDIA INFO, CLEAR OLD THUMBNAILs.

After a reboot and answering ENGLISH and I think one other prompt, I'm back. My FILE SOURCES are still defined. I'm betting once I get everything added back in it will still show which shows I've WATCHED. I don't get it. Some stuff stays no matter what.

[RoyalEF]

So after reboots because Boxee was hungup again and again, I finally reached the point where it finished identifying everything.

Without a doubt, having over a thousand adds to the database in a single source scan is a bad thing. If I had to start from scratch I would move most of my content out of the source folder into another directory (a fast move on most file systems). I'd add the source and let it identify maybe 250 movies. Then I'd move another 250 movies into the share and re-scan. Wait for it to finish identifying before adding the next bulk and repeating. Re-scans of "manageable" amounts of new adds to the db seems to go well--and fast.

After two days of pain with hangups to add over 7000 media files--rescans are NOT an issue. This mirrors my experience from my first purchase in july. Adding incrementally along the way was not a problem at all. It was only now, when I wiped the db and started from scratch that it was hopelessly hanging every hour of the identifying process. By dividing my tv shows into old and current, even that seemed to help. The switch to NFOs may affect speed -- I'm not done downloading the posters for all my TV shows. I want the posters to be local. I'll be losing my internet connection for 2-3 months.