On 2018 Jun 19 22:43:24, you wrote to me:

 >> AFAIK, seenbys and paths are not included in most dupe detection
 >> schemes... other non-changing control lines are fine to be included...
 >> one of the problems comes when some system sort those control lines on
 >> messages they are passing along... we don't see so much of that like we
 >> did at one time 

 RS> So some metadata is included in the data that is hashed for dupe
 RS> detection and some is not?

yes...

 RS> Are you sure about that?

yes... in fact, and i don't recall who pointed this out to me back in the '90s,
 dbridge does exactly this in a manner of speaking... it takes the whole
message header plus X bytes immediately following the message header and uses
all of that as at least part of the checksum calculation... this was pointed
out to me when i was working on my posting tool and was adding MSGID support to
 it...

i was using a library and just letting it do its thing... some of my test posts
 were reported as dupes when they clearly weren't... IIRC, they were detected
as dupes because they were posted within the same second... it turned out that
my MSGID was somewhere in the middle of the control lines at the beginning of
the message body and only my dbridge using testers were seeing this... someone
pointed out this thing about dbridge also using X bytes from the beginning of
the message body in addition to the message header so i moved my posting tool's
 MSGID to the top of the list and no more dupes were detected by those dbridge
systems...

i don't know what other systems do... there's only a very few that provide this
 information... SBBS is one of them... when i was testing Mystic, there was
some discussion about dupe detection as james worked to try to figure out the
best method he liked... i have used fastecho here for decades but i don't know
what data it uses for its checksums... i do know it uses two checksums,
though... i know this because i was being nosy one day and looking at FE's dupe
 database file (one for all message areas) with a hex viewer and noticed that
groups of bytes were repeated all throughout the file... i asked about this and
 was told i found a bug... basically, FE has two checksums that it uses for
each message and both are supposed to be stored in the database... what i found
 was that only one was being used and written to both fields... toby fixed that
 problem right quick... i just don't know what data is used to calculate
them...

back in the day, dupe detection formulas were not really shared around... maybe
 a couple of developers talking amongst themselves would tell each other what
they were doing but this information was not published where everyone could
find it... it was more or less black majik to a point...

 RS> Anyway, duplicate Message-IDs *should* be caught be any FTN software
 RS> written or updated in the past 20 years.

true... and we still have some systems that don't do MSGID at all so other
methods must be used on them...

)\/(ark

Always Mount a Scratch Monkey
Do you manage your own servers? If you are not running an IDS/IPS yer doin' it
wrong...
... WANTED: assistant to magician in beheading illusion. Blue Cross & salary.
---
 * Origin:  (1:3634/12.73)