Welcome to the MacNN Forums.

Uncle Skeleton · Sep 17, 2003, 11:24 AM

Imagine you have two files, file A and file B, each of which contain random chunks taken from file X. Is there a program that can merge A and B? like, go through A one kb at a time and if A has non-zero bytes where B has zero-bytes, copy the bytes from A into B, then save B as a new file (or even just save B)?

VEGAN · Sep 17, 2003, 11:59 AM

Originally posted by Uncle Skeleton:
Imagine you have two files, file A and file B, each of which contain random chunks taken from file X. Is there a program that can merge A and B? like, go through A one kb at a time and if A has non-zero bytes where B has zero-bytes, copy the bytes from A into B, then save B as a new file (or even just save B)?

Err... what to you mean "random chunks"?

If what you mean is that file X has been split in two: A and B; then there is a easy solution.

There is a `cat' command.

Open Terminal.app go to the folder where the two [or more] files are and execute this command

cat A B > X

Now you will have a file X!

If you have more than two files:

cat A B C D E > Y

Simple and easy [pay attention to the order of the files]

SUBGENIUS · Sep 17, 2003, 12:12 PM

What kind of files are we talking about?

Uncle Skeleton · Sep 17, 2003, 12:47 PM

no. X is 100 MB. A is 100 MB, but half of them are zeros, the other half are copied from X. B is 100 MB, but half (a different half than A, not contiguous, randomly distributed) are zeros and the rest are copied from X.

the idea is that you could be downloading the animatrix trailer (A) from bit torrent and it will get stuck halfway through. At the same time you're downloading the same file (B) from edonkey or fasttrack and it will get stuck halfway through. If you're lucky, the two halves will combine to make a whole (X), but even if not, the combination will be more complete than either A or B.

it mostly makes sense for files like MPEGs, where even partial downloads are better than nothing. But it should work for any download, assuming you intend to stick it out and wait for it to finish.

clear? any ideas?

VEGAN · Sep 17, 2003, 01:00 PM

the idea is that you could be downloading the animatrix trailer (A) from bit torrent and it will get stuck halfway through. At the same time you're downloading the same file (B) from edonkey or fasttrack and it will get stuck halfway through. If you're lucky, the two halves will combine to make a whole (X), but even if not, the combination will be more complete than either A or B.[/B]

BitTorrent can resume!!!

And, no, `cat' would not work with partial bittorrent files.

Uncle Skeleton · Sep 17, 2003, 01:06 PM

yes I know bit torrent can resume. But if you already have half the remaining data on your hard drive in File B, why waste your time and the time of those uploading to you and the time of those downloading (the parts in File B not in File A that have to share from the seeders, when they could be getting it from you)? you know? just trying to speed things along when the original sources from the two different networks are needlessly redundant? obviously some files download slower than others....

since it doesn't seem to be that complicated of a maneuver, I would assume a program already exists that is capable

edit: of course I'm not talking about popular files like the animatrix that it takes only hours to download. I'm talking about the svcds of Firefly and the odd episode of NewsRadio that it's taking me upwards of 50 days to finish. That's when it makes a difference

VEGAN · Sep 17, 2003, 01:25 PM

Originally posted by Uncle Skeleton:
yes I know bit torrent can resume. But if you already have half the remaining data on your hard drive in File B, why waste your time and the time of those uploading to you and the time of those downloading (the parts in File B not in File A that have to share from the seeders, when they could be getting it from you)?

Since YOU CAN RESUME you are not wasting NOBODY'S TIME... All it takes is the initial scan to see how much of the file you've got... and after that it's all go go go go...

Checking the file takes just a few seconds... unless you are talking about gigabites

Uncle Skeleton · Sep 17, 2003, 01:28 PM

it's the go go go part that takes a long time. I'm saying in some cases it would be faster (weeks faster) to scavange the data from other local files than it would to wait for it to download. And that time saved downloading is my time, the seeders' time, and the time of the people who could download those rare-er parts of the file from me instead of waiting in the queue for it from the seeders

VEGAN · Sep 17, 2003, 01:49 PM

Originally posted by Uncle Skeleton:
it's the go go go part that takes a long time. I'm saying in some cases it would be faster (weeks faster) to scavange the data from other local files than it would to wait for it to download. And that time saved downloading is my time, the seeders' time, and the time of the people who could download those rare-er parts of the file from me instead of waiting in the queue for it from the seeders

If you've got file A, partial file bittorrent DL, and file B, partial bittorent DL, you have already spent more time.

I guess what you are doing is having two simultaneous downloads... without understanding that you are downloading from A to B

Uncle Skeleton · Sep 17, 2003, 01:57 PM

It is not the plan from the start to open two simultaneous downloads of the same file. Of course that would be foolish. But what often happens is that in mlnet, I get identical search results from different networks (edonkey, bittorrent, fasttrack, etc). Not knowing which search result will be faster or start sooner, I choose them both. Days later, they each have downloaded part of the file (sometimes). Once I'm in this situation, it is sometimes faster to merge the two partials (or would be if it were possible) than to delete one (still not knowing which will finish first if left alone!) and just wait for the other one.

another situation would be that you and your housemate both go after the same file, then at 60% or so you realize it, and you both could shoot up to 90% or so if you could merge your two separate downloads.

I'm tired of discussing why I want to merge the files. Now that it is relatively clear what kind of merge I am talking about, can we try to think of ways it could be accomplished instead of reasons not to try?

mindwaves · Sep 17, 2003, 03:53 PM

E Pluribus Unum---> check Macupdate.com

Uncle Skeleton · Sep 17, 2003, 04:01 PM

I didn't find it on macupdate but I did on versiontracker. This program works the same as cat. it's not what I want.

here's what I have:

file A:

AAAAAAAA
00000000
AAAAAAAA
00000000

file B:

00000000
BBBBBBBB
00000000
BBBBBBBB

and I want to merge them to make this:

file X:

AAAAAAAA
BBBBBBBB
AAAAAAAA
BBBBBBBB

but what this program does is make this:

file AB:

AAAAAAAA
00000000
AAAAAAAA
00000000
00000000
BBBBBBBB
00000000
BBBBBBBB

get it?

thanks anyway though

Arkham_c · Sep 17, 2003, 04:20 PM

Writing such a program is pretty trivial. The question is what to do if byte 1023 in file a is 44 and in file b is 55? Do you declare it a loss at that point?

Uncle Skeleton · Sep 17, 2003, 04:33 PM

yeah, throw and error or warning and give up I think. That's another thing I would be interested to know from these files (that I can't tell from comparing them in HexEdit): are there any discrepancies between apartently identical downloads from different sources.

anyway, I've written a little utility since starting this thread that compares files one byte at a time, but since we can all agree it's a very simple functionality, I'm still hoping to find another implementation, hopefully one that uses larger buffers and some kind of optimization (altivec?), which are things that are beyond my abilities

Uncle Skeleton · Sep 17, 2003, 08:13 PM

ho hum, nevermind. I assumed the thing I wrote would take forever, because HexEdit takes forever and it's just comparing, not writing. But I tried mine on some large files and it's wicked fast. If anyone wants the source I'll post it, otherwise I won't clutter the page any more.

bad news is, I can't get the newly merged files to be recognized as a partial by mlnet (I don't have any actual torrent files, but I suspect since torrent actually does verify the partials that it would work). The reason I think it will work is because not long ago I had a years-old (before bit torrent et al arrived) mpeg file that I discovered had chunks missing. I went on mlnet and found the same file of the same size (but complete of course) and queued it up. Well lo and behold, mlnet recognized the old incomplete mpeg in my shared folder and copied it to the temp folder, and started downloading at 92%. As I said, I haven't gotten it to happen with the newly merged file, but I haven't yet tried deleting the partials and researching with the merged file in the shared folder. Anyway, life goes on...