visual basic alternative to tv_grab_rt

[leejsmith]

i have been working on a vb application to download epg data from the radiotimes website.

i tested this today and found 7 days of programs for a single channle including full descriptions , episode and if previously played downloaded in 30mins on my 150k broadband.

to use it you must have registerd and setup your channles on the RT website.

http://www.radiotimes.com

then you will have to register inet.ocx of the program gives you a missing ocx error.

just run regserv32 inet.ocx from a command prompt in the folder path that you unzipped the files.

then run the program.

at the top center of the screen is a button labled process and under that are 2 text boxes with a 4 and a 1.

the 4 is to start on the 4th date in the list (current date)
the 1 is the number of days to download.

it will open channles_ids for xmltv channle d's must be in the same directory at get_rt.exe
and output to a file called test.txt in xmltv format.
the only thing to do is open the test.txt and do a find and replace of & to + otherwise tv_sort gives an error.

then run tv_sort to sort and generate stop times plus make sure the file is ok for xmltv plugin.

here is the actual file

(i have removed the file until i have tested it some more.)

it looks messy as it still shows all my debug screen but at the moment i am looking for speed comparisons with tv_grab_uk_rt on other systems.

look forward to any questons / comments

Lee

[leejsmith]

speed test's for 1 day of bbc1

xmltv tv_grab_uk_rt -days=1 2.33 mins
program titles only

tv_grab_uk_rt xml file

xmltv tv_grab_uk_rt -days=1 -slow 5.40 mins
full program iinformation

tv_grab_uk_rt xml slow file

grab_rt 2.22 mins
program title / episode / repeat / descrption

grab_rt xml file

[leejsmith]

i left it running last night and downloaded 5 days x 32 channles in 4 hours.

there was a couple of errors running tv sort and i will look into them later.

[nielm]

The reason why XMLTV is slow is that it deliberatly ensures that there is up to 5 seconds between each webpage request.

The reason for this is to be nice to the server, preventing it from being overloading it with traffic, and thereby annoying to the admin, and causing him to notice that there is a spider leeching the information from his site (which is only on the borderline of legality), block the grabber or worse.

There have been several cases in the past of listings website owners noticing that something is spidering their pages (it is fairly obvious from the logs, when 1 IP address accesses several hundred pages in the space of a few minutes), and threatening legal action against both the the creators and users of the program.

[EDIT] for the above reason. and because you have to be a registed RT user to use your grabber, be careful about what email address you provide :-)

[leejsmith]

thanks for that i will look at adding a random pause between each net request.

the tv_sort errors are 99% fixed and the program is fully working.

will add the pause in a mo

i added a random pause plus the option to use a proxy server so it will be possible to hide your ip address with a anon proxy server.

will need to test this again for speed with the added delays.

[leejsmith]

with the random pause and using the anom proxy it runs slower than xmltv so i will continue to use xmltv for now.