Proposal: Automated mame rom management using peer to peer

Discussion:

(too old to reply)

poke49152

2003-10-28 00:21:52 UTC

Just an idea that I hope will someday come true:

Peer to peer is turning out to be a great way to distribute mame roms.
it would be wildly helpful if there was a cross referenced database of
rom zip CRCs and MD5s and a filesharing fingerprint such as a magnet
or ed2k URL that could be used by edonkey2000 or shareaza.

With this reference in place, a program like Clrmamepro could not only
identify bad/missing roms, but could fire off needed rom URLs to
shareaza or equivalent. This would allow you to complete your rom set
quickly and painlessly. All each user would have to do is share their
/roms folder on the shareaza network. Because each zip would be hashed
individually, updates would automatically be rehashed by shareaza, and
transfers would be rom by rom thus eliminating the transfer of giant
rar files that may contain hundreds of roms you already have.

To start this project off, all it would take is one person with a
complete current rom set to share it in sharaza and extract all of the
magnet and ed2k hashes for each file, list them on the web, and let
everyone else know about it. We could the logs from clrmame to
manually figure out which zips to go get, and as we each got the
complete set we could continue to share it. Hopefully someone like
Roman would write support in to a program like clrmamepro for the holy
grail of automated romset updating.

Thoughts?

poke49152

MCR

2003-10-28 10:28:52 UTC

Permalink

Sounds good in principal...

Post by poke49152
With this reference in place, a program like Clrmamepro could not only
identify bad/missing roms, but could fire off needed rom URLs to
shareaza or equivalent. This would allow you to complete your rom set
quickly and painlessly. All each user would have to do is share their
/roms folder on the shareaza network. Because each zip would be hashed
individually, updates would automatically be rehashed by shareaza, and
transfers would be rom by rom thus eliminating the transfer of giant
rar files that may contain hundreds of roms you already have.

Mmm...

Post by poke49152
To start this project off, all it would take is one person with a
complete current rom set to share it in sharaza and extract all of the
magnet and ed2k hashes for each file, list them on the web, and let
everyone else know about it. We could the logs from clrmame to
manually figure out which zips to go get, and as we each got the
complete set we could continue to share it. Hopefully someone like
Roman would write support in to a program like clrmamepro for the holy
grail of automated romset updating.

Now that sounds like a massive undertaking, in fact it sounds like it's
going to take too much time. Do you know how many ROMs there are? It
would consume too much time for me to do it, thats for sure.

I was under the impression clrmamepro could download from a source
location just by setting the URL. I used to do it with MAME.DK, but
haven't done it since. I assume it could do it with any URL, be it FTP
or whatever, better ask Roman that one.

Post by poke49152
Thoughts?

You wanna do the MD5 and CRC work for edonkey?

Post by poke49152
poke49152

--
MCR
MAME - History In The Making
I'm a burner. Full set on DVD
Email me for details.

poke49152

2003-10-28 12:45:54 UTC

Permalink

I would build the database of cross-referenced roms with ed2k hashes.
I found a tool that will generate ed2k hashes from a directory.

I figure the database would look like this:

8ball:
rom: 8b-cp1.bin [size: 2048] [CRC32: 0xc1f68754]
[SHA1:481c8e3dc35300f779b7925fa8a54320688dac54]
rom: 8b-cp2.bin [size: 2048] [CRC32: 0x6ec1d711]
[SHA1:768df8e621a7b110a963c93402ee01b1c9009286]
rom: 8b-cp3.bin [size: 2048] [CRC32: 0x4a9afce4]
[SHA1:187e5106aa2d0bdebf6ec9f2b7c2c2f67d47d221]
rom: 8b-gp1.bin [size: 4096] [CRC32: 0x1b4fb37f]
[SHA1:df6dd2766a3b70eec0bde0ae1932b35abdab3735]
rom: 8b-gp2.bin [size: 4096] [CRC32: 0xf193cdb5]
[SHA1:54fd1a10c1b9da0f9c4d190f95acc11b3c6e7907]
rom: 8b-gp3.bin [size: 4096] [CRC32: 0x191989bf]
[SHA1:dea129a4ed06aac453ab1fbbfae14d8048ef270d]
rom: 8b-gp4.bin [size: 4096] [CRC32: 0x9c64519e]
[SHA1:9a5cad7ccf8f1f289da9a6de0edd4e6d4f0b12fb]
rom: 8b-gp5.bin [size: 4096] [CRC32: 0xc50d0f9d]
[SHA1:31b6ea6282fec96d9d2fb74129a215d10f12cc9b]
rom: 8b-sdp1.bin [size: 4096] [CRC32: 0xa571daf4]
[SHA1:0db5b95db9da27216bbfa8fff84491a7755f9f1a]
rom: 8b-sp1.bin [size: 2048] [CRC32: 0x1ee167f3]
[SHA1:40c876a60832456a27108252ba0b9963f9fe70b0]

referenced to

ed2k://|file|8ball.zip|17407|7a7a60276e6efe9fc9e07a58f0cda81f|

That way you could search for an ed2k by rom SHA1. I could write a
parser that would take an uploaded clrmamepro log file and spit back
all of the ed2k urls you needed to complete your set. Almost like an
rsync update.

The weakness is that this depends on people continuing to share their
/roms folder. shareaza would be a great choice because it has separate
queues for large files (CHD's and stuff) and small files for the rest
of the roms. This way the idea wouldn't get plugged with people
getting CHD's--which in my opinion is gratuitous. I'm much more
interested in the older roms being an older guy :)

I could use some input on weaknesses that anyone sees with this scheme
before I go forward. One weakness would be that the server providing
the database (ie, my server) would have to always have the latest rom
set to be useful.

Also, if anyone know of a better place to discuss this than this NG
please let me know. It's hard to find the core MAME community. Could
someone point me to a good online resource that has hashes and
descriptions for the new releases as they come out? is there a new
mame.dk?

Post by MCR

Sounds good in principal...

Mmm...

Now that sounds like a massive undertaking, in fact it sounds like it's
going to take too much time. Do you know how many ROMs there are? It
would consume too much time for me to do it, thats for sure.
I was under the impression clrmamepro could download from a source
location just by setting the URL. I used to do it with MAME.DK, but
haven't done it since. I assume it could do it with any URL, be it FTP
or whatever, better ask Roman that one.

Post by poke49152
Thoughts?

You wanna do the MD5 and CRC work for edonkey?

Post by poke49152
poke49152

MCR

2003-10-28 13:00:01 UTC

Permalink

Post by poke49152
I would build the database of cross-referenced roms with ed2k hashes.
I found a tool that will generate ed2k hashes from a directory.

Excellent. The simpler the better. An automated program would make
things easier for everyone.

Post by poke49152
That way you could search for an ed2k by rom SHA1. I could write a
parser that would take an uploaded clrmamepro log file and spit back
all of the ed2k urls you needed to complete your set. Almost like an
rsync update.
The weakness is that this depends on people continuing to share their
/roms folder. shareaza would be a great choice because it has separate
queues for large files (CHD's and stuff) and small files for the rest
of the roms. This way the idea wouldn't get plugged with people
getting CHD's--which in my opinion is gratuitous. I'm much more
interested in the older roms being an older guy :)

This is the main problem with some in the MAME community. Not so long
ago I put most if not all the CHD's on Bittorrent, to enable everyone,
including dial up!!! users to download the sets. What I found was
people were downloading off the torrent and then closing their
connection and moving on, this way the available bandwidth for all was
still limited to me, then to the next person etc.

People are inherently selfish, when I checked my FTP server (not so long
ago) 3 out the 4 generic connections were help open with a 'NOOP'
(whatever that means) and no-one else was getting a look in. The last
remaining person (anonymous) was downloading a snap.zip file! While
it's there because I want to share it, those four tied up the connection
until I kicked them (not banned, they may have just left the machine on
while asleep).

Post by poke49152
I could use some input on weaknesses that anyone sees with this scheme
before I go forward. One weakness would be that the server providing
the database (ie, my server) would have to always have the latest rom
set to be useful.

People do lose interest in new schemes pretty quickly, unfortunately
thats the nature of the beast. This is why a few of us are offerring
FTP services, but they must all be at breaking point, my server has been
busy since I reset my machine. In the last four days my machine has
uploaded 16gigs of ROMs.

Post by poke49152
Also, if anyone know of a better place to discuss this than this NG
please let me know. It's hard to find the core MAME community. Could
someone point me to a good online resource that has hashes and
descriptions for the new releases as they come out? is there a new
mame.dk?

This is the perfect forum for it. Don't go to the official MAME board
unless you want a roasting.

--
MCR
MAME - History In The Making
I'm a burner. Full set on DVD
Email me for details.

Cerian Knight

2003-10-29 00:30:28 UTC

Permalink

MCR <***@tombstones.org.uk> gyrated his joystick in a pattern that spelled
out news:bnlp9i$vkufg$***@ID-208003.news.uni-berlin.de:

<snip>

Post by MCR
People are inherently selfish, when I checked my FTP server (not so long
ago) 3 out the 4 generic connections were help open with a 'NOOP'
(whatever that means) and no-one else was getting a look in. The last
remaining person (anonymous) was downloading a snap.zip file! While
it's there because I want to share it, those four tied up the connection
until I kicked them (not banned, they may have just left the machine on
while asleep).

That would explain why I had so much trouble getting in.
Just to let you know, I have never used a NOOP (No Operation) to keep a
connection alive while on your (or anyone elses) FTP. I sometimes start a
transfer and cancel it immediately if I feel my browsing is taking too
long, but very seldom.

I used to tick off my old ISP by leaving ICQ running when I was connected
so I wouldn't kick me off due to idle. They called me and told me I should
buy a permanent connection. So they had to explain to me exactly what
'they' meant by "Unlimited". I always suspected the word was ambiguous.

Being rude on the net as a matter of practice is, unfortunatly, all too
common. When one is too self-centered to see the bigger picture, that
behavior will follow. Same reason I only visit big cities, but would never
live in an American one (actually, I did live in Detroit for 2 years). At
least here on the net, when you lie with the dogs, the fleas keep their
distance.

Off to Newcastle, IN for a few days. Later
--
INC BX
INC BP
PUSH DX
DEC CX
INC CX
DEC SI

-----= Posted via Newsfeeds.Com, Uncensored Usenet News =-----
http://www.newsfeeds.com - The #1 Newsgroup Service in the World!
-----== Over 100,000 Newsgroups - 19 Different Servers! =-----

brandorf

2003-10-28 13:13:30 UTC

Permalink

Post by poke49152
Peer to peer is turning out to be a great way to distribute mame roms.
it would be wildly helpful if there was a cross referenced database of
rom zip CRCs and MD5s and a filesharing fingerprint such as a magnet
or ed2k URL that could be used by edonkey2000 or shareaza.
With this reference in place, a program like Clrmamepro could not only
identify bad/missing roms, but could fire off needed rom URLs to
shareaza or equivalent. This would allow you to complete your rom set
quickly and painlessly. All each user would have to do is share their
/roms folder on the shareaza network. Because each zip would be hashed
individually, updates would automatically be rehashed by shareaza, and
transfers would be rom by rom thus eliminating the transfer of giant
rar files that may contain hundreds of roms you already have.
To start this project off, all it would take is one person with a
complete current rom set to share it in sharaza and extract all of the
magnet and ed2k hashes for each file, list them on the web, and let
everyone else know about it. We could the logs from clrmame to
manually figure out which zips to go get, and as we each got the
complete set we could continue to share it. Hopefully someone like
Roman would write support in to a program like clrmamepro for the holy
grail of automated romset updating.
Thoughts?
poke49152

Interesting, but for this to work really well, I'd think that MAME
would need to switch to a rom naming convention similar to what the
GOOD Tools offers for Console emulators, because most P2P applications
use fuzzy searches, and the longer more informative filenames
(sometimes the CRC is in the filename) would ensure that you got the
'right' rom. A quick test with Kazaa and 'puckman', 'pacman', and
'pacmanm' all pretty much returns the same results, but something like
'Puckman (J) Rev 1' and 'Pacman (U) v1.0' or even 'Pacman
(crc-65c4f1c3)' would be more likely to return the correct file.
Granted, that would make command line mame a pain in the ass.

-Brandon K
-***@brandorf.com

2003-10-28 15:34:08 UTC

Permalink

I think this is a very good idea! The database containing the hashes can
also be used to generate web pages with the ed2k:\\ or magnet:\\ links
without actually hosting the files. If we add keywords, it can become
fully searchable.

I think this has a better shot at working the bittorrent because it just
involves sharing a folder and forgetting about it whereas bittorrent
requires you to leave the torrent open. Sharezaz provides great upload
bandwidth throttling to this wont hog all of peoples resources.

Count me in!